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SUMMARY 


The  objective  of  this  research  and  development  (RIO)  effort  was  to  develop  and  validate  an 
empirical  procedure  for  predicting  training  outcomes  associated  with  Initial-skills  courses 
administered  by  the  United  States  Air  Force  Air  Training  Coenand.  The  RIO  consisted  of  the 
collection  and  analysis  of  information  to  determine  the  effects  of  student  attributes  and  course 
properties  on  training  outcomes  such  as  academic  training  performance  and  student  elimination. 
The  value  of  the  procedure  resides  in  its  use  as  a  decision  support  system  for  personnel  and 
training  management.  It  can  be  applied  on  a  course-by-course  basis  to  evaluate  "what-if* 
questions  concerning  the  effects  of  changes  in  student  and  course  characteristics  on  training 
outcomes.  For  example,  the  system  could  be  used  by  personnel  managers  to  forecast  the  training 
outcomes  likely  to  result  from  decreases  in  the  minimum  aptitude  test  score  required  for  entry 
into  an  initial-skills  course.  It  could  be  used  by  training  managers  to  forecast  the  training 
outcomes  likely  to  result  from  increases  or  decreases  in  the  trained  personnel  requirement  for 
the  course.  It  could  be  used  by  course  designers  to  forecast  the  training  outcomes  likely  to 
result  from  adjustments  in  course  length  or  student/instructor  ratio.  In  addition  to  providing  a 
means  of  forecasting  training  outcomes  associated  with  changes  in  single  variables,  it  could  be 
used  to  forecast  training  outcomes  associated  with  simultaneous  adjustments  in  several  student 
and  course  characteristics.  The  payoff  of  this  RID  is  a  decision  support  system  for  personnel 
and  training  management  that  can  be  applied  to  anticipate  and  avoid  adverse  training  outcomes. 
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PREFACE 


This  R&D  is  one  of  several  efforts  conducted  in  support  of  Request  for  Personnel 
Research  (RPR)  73-17,  Minimum  Aptitude  Requirements  for  Air  Force  Enlisted  Specialties. 
The  primary  objective  of  research  conducted  in  support  of  the  RPR  was  to  develop  an 
empirical  procedure  for  evaluating  aptitude  requirement  mlnimums  for  enlisted 
specialties.  Such  a  procedure  was  developed  and  is  described  in  an  AFHRL  technical 
report  by  Burtch,  Lipscomb,  &  Wlssman  (1982)  and  a  special  report  by  Weeks  (1984).  The 
RtO  described  in  this  report  extends  the  focus  of  the  research  to  include  development  of 
a  system  for  assessing  the  impact  of  aptitude  requirement  adjustments  on  initial-skills 
training.  The  emerging  system  is  potentially  useful  not  only  for  avoiding  the  adverse 
effects  on  training  outcomes  of  changes  in  student  aptitudes  but  also  for  avoiding  the 
adverse  effects  of  changes  in  initial-skills  courses.  The  research  was  accomplished 
under  work  unit  7719-19-10  and  supports  the  Manpower,  Personnel,  and  Training 
Integration  subthrust  and  the  Manpower  and  Force  Management  thrust. 

Credit  is  due  the  Air  Force  Manpower  and  Personnel  Center,  Directorate  of 
Assignments,  Skills  Management  Division,  USAF  Classification  and  Training  Branch 
(AFMPC/DPMRTC) ,  Randolph  AFB,  TX,  for  support  of  the  project  under  RPR  73-17.  The 
research  could  not  have  been  conducted  without  the  generous  support  of  the  Air  Training 
Command  (ATC).  Resources  provided  by  ATC  and  authorization  to  conduct  the  RID  are 
documented  in  ATC  Training  Research  Agreements  #120  and  #136.  Because  so  many 
individuals  participated  in  the  project,  space  limitations  do  not  permit  mentioning  them 
all  by  name.  Training  personnel  located  at  each  of  the  following  organizations 
participated  in  the  project:  Air  Force  Military  Training  Center  (AFMTC),  Lackland  AFB, 
TX;  the  3700th  Technical  Training  Wing  (TCHTW),  Sheppard  AFB,  TX;  the  USAF  School  of 
Health  Care  Sciences,  Sheppard  AFB,  TX;  the  3400th  TCHTW,  Lowry  AFB,  CO;  the  3330th 
TCHTW,  Chanute  AFB,  IL;  and  the  3300th  TCHTW,  Keesler  AFB,  MS.  Credit  is  especially  due 
course  instructors  and  instructor  supervisors  for  taking  time  from  their  busy  schedules 
to  provide  information  during  initial  interviews.  Credit  is  also  due  personnel  of  the 
registrar's  office  at  each  training  wing  for  their  assistance  in  obtaining  student 
training  records.  Finally,  credit  is  due  Headquarters  ATC,  Deputy  Chief  of  Staff 
Technical  Training,  Plans  Directorate,  Research  and  Technology  Division  (HQ  ATC/TTXR) 
for  assistance  in  coordinating  the  data  collection  effort. 

This  report  has  been  produced  while  RIO  continues  under  work  unit  7719-19-10  for 
the  purpose  of  developing  a  computer-based  system  to  facilitate  application  of  the 
model.  User-friendly  software  which  Implements  the  various  parameters  of  the  model  has 
been  developed  under  the  acronym  TTIM  (Technical  Training  Impact  Model).  This  effort 
will  be  documented  In  subsequent  reports.  In  addition,  final  test  and  evaluation 
efforts  are  being  coordinated  to  serve  as  a  basis  of  transitioning  the  technology  to  HQ 
ATC. 
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AN  EMPIRICAL  SYSTEM  FOR  ASSESSING  THE  IMPACT 
OF  APTITUDE  REQUIREMENT  ADJUSTMENTS  ON  AIR  FORCE 
INITIAL  SKILLS  TRAINING 


I.  INTRODUCTION 


Background 

Although  many  factors  Influence  the  effectiveness  of  an  organization,  it  has  long  been 
recognized  that  individual  performance  is  one  of  the  most  important  determinants  of  goal 
attainment  in  any  group  undertaking.  As  a  result,  a  wide  variety  of  studies  have  been  conducted 
in  an  attempt  to  delineate  the  variables  that  Influence  Individual  performance,  and  to  determine 
how  these  variables  Interact  with  situational  constraints  to  influence  organizational  effective¬ 
ness.  Among  the  variables  examined  in  these  studies,  the  allocation  of  individuals  to  jobs  has 
proven  to  be  of  special  significance,  due  to  the  fact  that  individuals  differ  markedly  in  their 
willingness  and  capability  to  perform  various  tasks  (Dunnette,  1966).  For  instance,  Schmidt, 
Hunter,  McKenzie,  and  Muldrow  (1979)  have  shown  that  a  single  organization  may  save  itself 
millions  of  dollars  each  year  by  assigning  to  jobs  only  those  individuals  who  possess  the 
attributes  necessary  to  satisfactorily  perform  those  jobs. 

Recognition  of  the  fundamental  importance  of  optimal  manpower  allocation  has  led  many 
organizations  to  Invest  substantial  time  and  energy  In  the  development  of  decision  rules  for 
assigning  Individuals  to  jobs.  Typically,  this  Is  accomplished  through  the  use  of  standard 
selection  Inventories  to  assess  the  extent  to  which  an  individual  possesses  the  attributes 
required  for  adequate  job  performance.  However,  in  other  instances,  it  has  been  necessary  to 
employ  a  more  general  and  complex  classification  system  in  which  the  Individual  Is  considered  for 
assignment  to  a  number  of  alternate  jobs  on  the  basis  of  his/her  attributes  and  organizational 
needs  (Magnuson,  1966).  Application  of  the  more  complex  classification  paradigm  for  manpower 
allocation  has  generally  been  found  to  be  most  useful  when  the  organization  must  obtain  adequate 
performance  in  a  large  number  of  jobs  within  a  finite  timeframe  and  manpower  pool. 

The  Air  Force  Is  perennially  faced  with  conditions  that  underscore  the  need  for  effective 
classification.  Because  Air  Force  technical  training  is  expensive  and  time-consuming,  and 
because  It  constitutes  a  hurdle  that  most  airmen  must  pass  before  they  can  be  assigned  to  Air 
Force  specialties  (AFSs),  classification  decisions  have  been  based.  In  part,  on  expected  training 
success.  In  this  regard.  It  should  be  noted  that  effective  allocation  of  Individuals  with 
respect  to  job  training  programs  Is  not  associated  solely  with  short-term  benefits.  As  Maglnnis, 
Uchlma,  and  Smith  (1975a,  1975b,  1975c)  have  pointed  out,  inappropriate  allocation  of  Air  Force 
enlistees  to  training  programs  may  result  in  a  number  of  costly  long-term  outcomes  ranging  from 
poor  job  performance  to  high  turnover.  When  these  negative  outcomes  of  inappropriate  job 
classification  decisions  are  aggregated  over  tens  of  thousands  of  individuals,  they  can  have  a 
substantial  Impact  on  Air  Force  efficiency  and  mission  readiness. 

Personnel  Allocation  In  the  Air  force 

Given  the  foregoing  concerns,  it  is  hardly  surprising  that  a  cohesive  classification  strategy 
has  emerged  in  the  Air  Force.  As  Implied  by  the  previous  discussion,  any  effective  personnel 
allocation  system  requires  two  central  pieces  of  Information:  (a)  a  definition  of  the 
performance-relevant  attributes  possessed  by  the  individual,  and  (b)  a  system  for  relating 
personnel  attributes  to  effective  job  performance. 

In  the  Air  Force,  attributes  of  the  individual  are  assessed  primarily  through  the  Armed 
Services  Vocational  Aptitude  Battery  (ASVAB)  (Weeks,  Mullins,  t  Vitola,  1975).  The  ASVAB  Is  a 


cognitive  abilities  test  battery  composed  of  a  set  of  aptitude  tests  ranging  front  general  verbal 
and  quantitative  measures  to  specialized  measures  of  technical  knowledge.  The  Air  Force  combines 
the  tests  contained  in  the  ASVA8  to  form  four  aptitude  test  composites.  These  four  aptitude 
composites  are  designed  to  measure  an  individual's  aptitude  for  Mechanical.  Administrative, 
General,  and  Electronics  job  specialty  areas.  The  ASVAB  aptitude  composites  display 
reliabilities  in  the  mid-. 80s  and  validities  against  a  criterion  of  technical  training  success  in 
the  mid-. 60s  (Weeks,  Mullins,  &  VI tola,  1975).  The  scores  on  the  aptitude  composites  constitute 
the  principal  measures  employed  by  the  Air  Force  for  describing  characteristics  of  the  individual 
likely  to  influence  subsequent  training  and  job  performance. 

Although  ASVAB  aptitude  composites  constitute  the  principal  description  of  the  individual 
used  in  the  Air  Force  manpower  personnel  allocation  system,  other  descriptors  are  employed  as 
well.  For  Instance,  the  educational  level  and  the  educational  preparation  of  the  enlistee  are 
considered,  as  are  expressed  Interest  for  a  given  specialty  and  the  individual's  physical 
capabilities.  In  addition,  access  to  certain  specialties  may  be  limited  by  manpower  requirements 
and  administrative  requirements,  such  as  obtaining  a  security  clearance. 

The  system  the  Air  Force  employs  for  relating  performance  on  the  ASVAB  to  organizational 
needs  is  relatively  straightforward.  Essentially,  it  is  based  on  the  concept  that  the  most 
talented  individuals  should  be  assigned  to  the  most  demanding  AFSs.  In  the  Air  Force  context, 
talent  for  various  specialties  is  defined  in  terms  of  the  individual's  scores  on  the  ASVAB 
aptitude  composites.  Thus,  under  ideal  conditions,  individuals  with  high  scores  on  the 
Electronics  composite  would  be  assigned  to  the  most  difficult  electronics  specialties.  One  way 
the  Air  Force  has  attempted  to  implement  this  assignment  rule  is  by  specifying  minimun  aptitude 
requirements  for  entry  into  each  AFS. 

For  the  personnel  allocation  strategy  described  above  to  serve  its  intended  purpose,  aptitude 
requirement  minimums  must  correspond  to  the  occupational  demand  or  difficulty  of  job 
specialties.  In  setting  aptitude  requirement  minimums,  technical  training  attrition  rates  have 
served  as  an  index  of  difficulty.  Although  this  index  of  difficulty  appears  reasonable,  changes 
in  training  programs  and  manpower  requirements,  as  well  as  more  general  administrative  concerns, 
have  led  to  changes  in  technical  training  attrition  rates  and,  therefore,  aptitude  requirement 
minimums  that  are  not  necessarily  in  line  with  the  difficulty  of  the  job.  For  example.  Weeks 
(1984)  found  a  less-than-perfect  relationship  between  the  aptitude  minimums  assigned  AFSs  and 
experts'  judgments  of  job  difficulty.  Because  the  optimal  allocation  of  talent  depends  on  a 
close  match  between  aptitude  requirement  minimums  and  the  relative  difficulty  of  job  specialties, 
misalignments  between  aptitude  requirements  and  job  difficulty  lead  to  assignment  errors,  and  to 
corresponding  effects  on  Air  Force  mission  readiness. 


Defining  Occupational  Difficulty 

The  definition  of  occupational  difficulty  represents  a  salient  problem.  First,  it  is  clear 
that  training  attrition  rates  can  produce  only  an  approximate  measure  of  difficulty  since  they 
reflect  training  requirements  as  opposed  to  job  requirements.  Second,  attrition  rates  are 
Influenced  by  a  variety  of  training-specific  factors  (such  as  the  appropriateness  of 
instructional  materials  or  the  availability  of  faculty  resources)  that  may  result  in  a  biased 
index  of  difficulty.  Thus,  it  appears  that  in  implementing  its  manpower  allocation  system  in  the 
most  effective  manner  possible,  the  fundamental  problem  facing  the  Air  Force  is  to  find  a  more 
appropriate  measure  of  relative  occupational  difficulty. 

The  fundamental  import  of  this  problem  and  its  potential  impact  on  the  overall  effectiveness 
of  the  Air  Force,  via  its  implications  for  personnel  allocation,  led  to  the  requirement  for  the 
Air  Force  Human  Resources  Laboratory  (AFHRL)  to  undertake  an  extensive  series  of  research  and 
development  (RID)  efforts  intended  to  establish  a  more  general  and  accurate  measure  of  the 


relative  difficulty  of  enlisted  job  specialties.  Although  there  are  obviously  many  potential 
obstacles  Involved  in  attempting  to  assess  the  difficulty  of  any  career  field,  we  believe  that 
AFHRt  has  formulated  a  viable  approach  to  this  problem  through  the  construct  of  occupational 
learning  difficulty. 

Occupational  learning  difficulty  Is  defined  as  the  time  it  takes  to  learn  to  perform  an 
occupation  satisfactorily  (Weeks,  1984).  The  construct  is  based  on  the  premise  that,  with 
ability  held  constant,  individuals  will  take  a  longer  time  to  learn  to  perform  a  more  difficult 
occupational  task  than  a  relatively  easy  one  (Christal,  1976).  This  appears  to  be  a  well-founded 
assumption.  In  fact,  substantial  support  for  this  proposition  may  be  found  in  the  research 
literature  with  respect  to  its  corollary  that  more  capable  individuals  will  take  less  time  to 
learn  a  task  of  constant  difficulty  (Cronbach  &  Snow,  1977).  Moreover,  the  recent  cognitive 
processing  literature  also  strongly  supports  the  proposition  that  time-to-mastery  and  response 
time  may  be  excellent  indicators  of  problem  difficulty  at  both  the  Individual  and  the  aggregate 
levels  (Sternberg,  1982).  Thus,  It  appears  that  there  is  sufficent  evidence  in  the  literature  to 
indicate  that  the  time  taken  by  an  individual  to  learn  a  task  is  an  appropriate  definition  of 
difficulty. 

If  It  is  granted  that  "time  to  learn"  provides  an  adequate  index  of  the  difficulty  of  an 
activity,  then  the  next  question  to  be  answered  is  how  this  conceptualization  of  difficulty  might 
be  translated  into  an  operational  measure.  In  addressing  this  problem,  the  Air  Force  employed  a 
job  analysis  approach  in  which  the  time  it  takes  to  learn  to  satisfactorily  perform  occupational 
tasks  is  evaluated  by  subject-matter  experts  (Christal,  1976).  .  More  specifically,  AFHRL 
developed  a  set  of  benchmark  rating  scales  against  which  the  learning  difficulty  of  the  tasks 
performed  in  specialties  within  the  Mechanical,  Administrative,  General,  and  Electronics  job 
specialty  areas  could  be  evaluated  (Fugill,  1972,  1973).  Subsequently,  experts  evaluated  tasks 
within  each  specialty  against  the  appropriate  benchmark  rating  scale,  and  a  weighted  composite  of 
these  task  ratings  was  obtained  to  determine  the  overall  learning  difficulty  of  the  specialty 
(Weeks,  1981).  Research  has  shown  that  these  task  ratings  are  highly  reliable  and  display 
considerable  convergent  validity  (Burtch,  Lipscomb,  &  Wissman,  1982). 

The  foregoing  investigations  indicated  that  it  is  possible  to  employ  ratings  of  the  learning 
difficulty  of  occupational  tasks  In  evaluating  overall  occupational  difficulty.  Moreover,  this 
research  demonstrated  that  measures  of  occupational  learning  difficulty  could  be  obtained  through 
a  relatively  straightforward  extension  of  the  Air  Force's  current  job  analysis  system  (Christal, 
1976).  Consequently,  It  appeared  that  the  occupational  difficulty  measure  might  provide  a  fully 
adequate  and  administratively  feasible  basis  for  defining  the  relative  difficulty  of 
occupations.  If  this  measure  were  In  turn  used  as  a  frame  of  reference  in  establishing  job 
aptitude  requirement  minlmums,  it  was  felt  that  it  might  improve  the  current  personnel  allocation 
system.  However,  preliminary  analyses  (Weeks,  1984)  have  indicated  that  the  realignment  of 
aptitude  mlnimums  In  accordance  with  occupational  learning  difficulty  would  lead  to  marked  shifts 
In  aptitude  minlmums  for  several  specialties. 


The  Problem 

A  wide  variety  of  studies  have  indicated  that  scores  on  aptitude  measures  are  directly 
related  to  educational  performance.  For  instance,  Tyler  (1964)  summarized  a  number  of  studies 
and  concluded  that  general  cognitive  ability  has  a  marked  impact  on  training  performance,  such 
that  higher  aptitude  students  not  only  perform  better  on  achievement  tests  but  also  seem  to 
profit  more  from  education.  Similarly,  in  the  Air  Force  context,  Wilbourn,  Valentine,  and  Ree 
(1984)  found  that  performance  on  the  ASVA8  is  substantially  related  to  training  performance  In  a 
number  of  different  occupational  fields. 


Obviously,  intellectual  aptitude  is  not  the  only  variable  that  can  influence  training 
performance.  For  instance,  it  has  been  pointed  out  that  interest,  general  achievement 
motivation,  specific  educational  preparation,  reading  ability,  maturity,  study  habits,  and 
self-discipline,  among  other  variables,  may  also  affec1.  training  performance  (Frederickson  6 
Gilbert,  1964;  Gough  &  Fink,  1964).  Moreover,  as  Terman  v 1 959)  pointed  out,  all  these  variables 
interact  with  general  cognitive  ability  in  subtle  but  generally  positive  ways.  As  a  result.  It 
is  critical  that  any  effort  to  align  aptitude  requirement  minimums  with  occupational  learning 
difficulty  include  careful  consideration  of  the  impact  of  such  aptitude  requirement  adjustments 
on  technical  training  outcomes.  Further,  one  must  consider  not  only  the  direct  effects  of 
aptitude  but  also  the  indirect  effects  that  aptitude  has  on  other  individual  attributes  related 
to  training  performance. 

In  considering  the  potential  impact  of  aptitude  requirement  adjustments  on  training 
performance,  there  is  another  set  of  variables  that  should  be  considered.  Currently,  the  Air 
Force  formulates  instructional  programs  through  a  variation  of  the  instructional  systems 
development  ( ISD)  paradigm  proposed  by  Goldstein  (1974)  and  elaborated  by  O'Neil  (1979)  among 
others.  As  with  most  well-designed  instructional  systems,  the  characteristics  of  the  student 
population  are  carefully  examined,  including  educational  preparation,  reading  ability,  aptitude, 
and  motivation.  Thus,  the  indirect  impact  of  aptitude  requirement  adjustments  on  technical 
training  might  be  manifested  not  only  through  other  attributes  of  the  individual,  but  also 
through  the  design  of  training  materials  and  course  variables  such  as  the  length  of  training. 

It  should  also  be  recognized  that  the  impact  of  aptitude  on  training  performance  might  itself 
be  moderated  by  course  content  variables,  and  that  aptitude  may  Interact  with  these  variables  in 
determining  training  outcomes.  For  instance,  Cronbach  and  Snow  (1977)  reviewed  different  sources 
of  evidence  that,  while  not  conclusive,  do  suggest  that  features  of  the  instructional  process  may 
interact  with  aptitude  and  other  individual  attributes  in  determining  training  outcomes.  It 
should  be  noted  here  that  similar  effects  have  been  observed  in  the  military  setting  in  studies 
conducted  by  Fox,  Taylor,  and  Caylor  (1969),  as  well  as  Federico  and  Landis  (1979). 

Taken  as  a  whole,  the  preceding  discussion  indicates  that  changes  in  aptitude  requirement 
minimums  will  lead  to  changes  in  training  performance.  However,  it  is  difficult  to  specify  the 
exact  nature  and  magnitude  of  these  changes  since  the  effects  of  aptitude  may  be  moderated  by 
other  individual  attributes  and  by  various  characteristics  of  the  particular  training  program. 

This  ambiguity  was  of  sufficient  concern  to  Air  Force  training  and  personnel  managers  that  It 
was  felt  that  before  any  major  adjustments  were  made  in  aptitude  requirements  on  the  basis  of 
occupational  learning  difficulty,  RiO  should  be  conducted  to  provide  some  insight  into  the  Impact 
of  such  aptitude  requirement  adjustments  on  technical  training  outcomes.  Finally,  it  was  argued 
that  the  investigation  should  carefully  consider  those  complex  attributes  of  both  students  and 
training  programs  that  might  Interact  with  aptitude  in  determining  training  outcomes.  The 
present  investigation  was  undertaken  in  an  attempt  to  meet  these  goals. 


II.  METHOD 


Overview 


A  variety  of  different  methodological  strategies  might  be  employed  to  address  the  concerns 
outlined  previously.  However,  the  nature  of  Air  Force  initial-skills  training,  along  with  the 
goals  of  the  present  effort  and  the  pertinent  research  literature,  indicated  the  need  for  a 
particular  methodological  and  conceptual  approach.  In  reviewing  the  relationship  between 
cognitive  abilities  and  training  outcomes,  it  became  apparent  that  the  effects  of  aptitude  on 
training  outcomes  could  adequately  be  captured  only  by  considering  a  variety  of  student 


attributes  related  to  aptitude,  such  as  academic  motivation  and  educational  level,  as  well  as  the 
potential  interaction  of  aptitude  and  associated  variables  with  aspects  of  the  training  process 
associated  with  a  given  specialty.  Moreover,  Kirkpatrick  (1959)  noted  that  multiple  training 
outcome  variables  must  be  examined  in  any  attempt  to  assess  training  performance.  For  example, 
when  examining  technical  training  outcomes,  it  is  clear  that  the  quality  of  student  performance 
must  be  considered  as  well  as  student  attrition  rates.  These  observations  taken  as  a  whole 
suggest  that  any  attempt  to  address  the  issues  at  hand  must  take  a  multivariate  approach  capable 
of  incorporating  a  variety  of  student  input,  course  content,  and  training  outcome  variables. 

In  addition  to  a  multivariate  approach,  practical  considerations  require  that  the 
relationships  established  among  these  variables  must  be  capable  of  being  used  in  making  routine 
predictions.  This  requires  that  measures  of  these  variables  be  applicable  to  all  technical 
training  courses  under  consideration.  When  this  observation  was  coupled  with  the  fact  that  R&D 
examining  training  outcomes  would  be  feasible  only  to  the  extent  that  it  did  not  disrupt  ongoing 
training  efforts,  a  correlational  field  study  seemed  the  most  appropriate  approach.  Finally, 
because  the  Air  Force  Air  Training  Command  (ATC)  currently  administers  approximately  200 
resident,  initial-skills  training  courses,  it  was  apparent  that  the  student  input,  course 
content,  and  training  outcome  variables,  as  well  as  their  interrelationships,  could  not  be 
established  separately  for  each  course.  Thus,  it  seemed  necessary  to  employ  general  descriptive 
variables  in  each  of  these  areas  and  attempt  to  establish  typical  relationships  across  a  variety 
of  initial-skills  courses. 

In  view  of  these  considerations,  it  was  decided  to  use  the  modeling  strategy  of  path 
analysis,  which  is  an  extension  and  generalization  of  the  traditional  regression  model  and  which, 
like  the  regression  approach,  allows  predictions  in  such  a  way  that  permits  multiple  causal 
relationships  to  be  considered.  This  decision  was  supported  by  the  fact  that  multivariate  models 
have  proven  useful  in  addressing  a  variety  of  complex  problems  in  the  social  sciences,  ranging 
from  interest  development  (Ekehammar,  1977)  to  the  determinants  of  collegiate  academic 
performance  (Ribich  &  Schneck,  1979).  An  excellent  overview  of  this  diverse  literature,  as  well 
as  illustrations  of  the  potential  applications  of  this  approach  in  addressing  problems  involving 
multivariate  prediction,  may  be  found  in  Joreskog  and  Sorbom  (1980). 

After  it  was  determined  that  a  multivariate  modeling  approach  would  be  most  appropriate  for 
addressing  the  problems  at  hand,  the  next  issue  to  be  resolved  was  how  this  approach  should  be 
implemented.  Generally  speaking,  the  use  of  a  multivariate  modeling  approach  is  contingent  on 
the  formation  of  a  sound  theoretical  framework  (Kenny,  1979).  This  theoretical  framework  in  turn 
serves  as  a  guide  for  determining  the  nature  of  the  variables  that  should  be  included  in  the 
model,  as  well  as  their  hypothetical  interrelationships.  The  model  is  then  tested  by  contrasting 
theoretical  expectations,  and  the  variance-covariance  matrix  they  imply,  against  an  empirical 
variance-covariance  matrix  derived  from  a  set  of  observations.  If  the  theoretical  model  fits  the 
data,  one  can  have  confidence  in  its  appropriateness.  The  weights  generated  in  fitting  the 
theoretical  model  to  the  observed  data  may  then  be  used  as  a  basis  for  prediction. 

The  foregoing  overview  of  the  path  analysis  served  to  outline  the  general  methodological 
steps  employed  in  the  present  investigation.  First,  a  general  conceptual  framework  for 
understanding  the  relationships  among  student  inputs,  course  content,  and  training  outcomes  was 
formulated.  Second,  this  theoretical  framework  guided  the  specification  of  the  variables  likely 
to  influence  training  performance.  Third,  adequate  measures  of  these  variables  were  developed  or 
obtained.  Fourth,  the  measures  were  applied  to  a  number  of  different  individuals  in  a 
representative  sample  of  initial-skills  courses.  Fifth,  the  observed  relationships  among  these 
variables  were  evaluated  in  light  of  the  model.  Of  course,  modifications  were  made  to  this 
general  methodological  framework  as  a  result  of  practical  constraints.  For  instance. 
Investigators  commonly  modify  their  initial  theory  once  correlational  data  become  available  if 


any  limitations  in  their  original  theory  become  apparent.  Nevertheless,  these  steps  constitute 
the  general  method  employed  in  the  present  investigation. 


The  Conceptual  Model 


The  preceding  section  suggested  some  ways  aptitude  can  influence  performance  in  training. 
This  section  will  develop  this  relationship  in  more  detail,  starting  with  the  most 
straightforward  relationship,  which  assumes  that  aptitude  directly  causes  training  performance. 
Model  A  of  Figure  1  schematically  depicts  this  relationship.  Although  this  may  be  an  appropriate 
conceptualization  in  situations  where  the  aptitude  measures  are  heavily  weighted  toward  prior 
achievement,  in  the  context  of  the  present  effort,  it  is  unlikely  that  such  a  simple,  direct 
relationship  exists,  since  the  ASVAB  is  a  general  measure  of  intellectual  ability  and  most  Air 
Force  enlistees  have  little  prior  experience  with  the  work  required  in  the  specialties  for  which 
they  are  trained. 
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Figure  1.  Initial  Conceptual  Models. 


These  observations  suggested  that  some  fundamental  intervening  processes  must  link  aptitude 
to  training  performance.  There  can  be  little  doubt  that  this  Intervening  process  is  learning. 
This  extension  of  the  simple,  direct  linkage  model  is  presented  in  Figure  1  as  Model  B.  In  this 
case,  aptitude  facilitates  ease  of  learning,  which  in  turn  facilitates  training  performance, 
given  a  constant  training  time.  This  extension  of  the  initial  model  is  supported  In  a  literature 
review  by  Cronbach  and  Snow  (1977),  who  sumnarized  a  number  of  studies  indicating  that  aptitude 
generally  leads  to  more  rapid  attainment  of  training  performance  standards.  Moreover,  additional 
support  for  the  extended  model  is  suggested  by  its  alignment  with  the  theory  underlying  the 
occupational  learning  difficulty  research  conducted  by  AFHRL. 


When  ease  of  learning  is  conceived  of  as  a  hypothetical  variable  Intervening  between  aptitude 
and  performance,  the  significance  of,  and  need  for,  a  second  class  of  variables  becomes 
apparent.  One  of  the  principles  underlying  instructional  systems  development  Is  that  performance 
in  training  can  be  enhanced  by  the  use  of  effective  training  strategies  (Goldstein,  1974).  This 
suggests  that  variables  that  characterize  the  training  program  and  that  might  influence  ease  of 


learning,  such  as  quality  of  instruction  and  amount  of  feedback  (Latham  &  Wexly,  1980),  should  be 
Included  in  the  model.  Model  C  presents  a  revision  of  Model  B  which  incorporates  training 
variables.  However,  it  should  be  recognized  that,  as  well  as  having  an  impact  on  ease  of 

learning,  these  course  content  variables  may  also  have  a  separate  Impact  on  performance.  For 
example,  instructor-student  interactions  may  affect  evaluations  of  student  performance,  in 
addition  to  having  an  impact  on  ease  of  learning.  Thus,  it  was  necessary  to  consider  the 

possibility  that  course  content  variables  may  have  a  direct  Impact  on  performance.  Moreover,  it 
should  be  noted  that  because  initial-skills  training  courses  are  designed  in  relation  to  the 
student  population,  aptitude  Itself  may  also  have  a  direct  Impact  on  these  course  content 
variables.  These  considerations  led  to  Model  0. 

Support  for  Model  D  is  also  found  in  the  literature.  This  model  postulates  a  complex  set  of 
interactions  between  aptitude  and  course  content  variables  and  holds  that  both  classes  of 

variables  m^y  have  direct  effects  on  performance.  Phenomena  of  this  sort  have  been  observed  in  a 

variety  of  studies  including  those  conducted  by  Smith  (1942)  and  Newman,  Freeman,  and  Holzinger 

(1937).  Yet,  despite  the  existence  of  such  support,  this  version  of  the  conceptual  model  is 

still  incomplete.  The  literature  reviewed  earlier  suggests  that  any  effects  of  aptitude  on 
training  performance  may  be  expressed  through  or  moderated  by  other  student  inputs  such  as 
reading  ability  or  academic  motivation,  which  are  themselves  related  to  aptitude.  As  a  result, 

it  seems  that  the  aptitude  component  of  the  model  should  be  expanded  to  include  a  variety  of 

performance-relevant  attributes  of  the  individual.  However,  it  should  be  recognized  that 

variables  such  as  motivation  may  themselves  have  a  direct  impact  on  performance  which  is 

independent  of  their  impact  on  ease  of  learning.  Finally,  it  should  be  noted  that  although 
training  performance  represents  an  abstract  construct,  it  is  manifested  in  a  variety  of  training 
outcomes  which,  in  some  way,  have  cost  implications  for  the  Air  Force. 

The  foregoing  considerations  led  to  a  further  revision  of  the  initial  conceptual  model.  This 
model,  presented  in  Figure  2,  represents  the  general  conceptual  model  employed  in  the  present 
investigation.  As  may  be  seen,  both  course  content  variables  and  student  inputs  were  held  to 
have  positive  causal  effects  on  training  outcomes,  as  indicated  by  the  plus  signs  and  arrows. 
Student  input  and  course  content  variables  were  also  held  to  be  capable  of  having  a  direct  causal 
effect  on  training  outcomes  regardless  of  ease  of  learning.  Finally,  ease  of  learning  was 

assumed  to  have  a  direct  positive  causal  effect  on  training  outcomes,  and  adverse  training 

outcomes  were  assumed  to  have  a  negative  effect  on  training  costs.  Basically,  the  general 
conceptual  model  Indicates  that  training  outcomes  are  a  function  of  student  inputs  and  course 
content,  and  that  these  outcomes  in  turn  affect  training  costs.  This  rather  straightforward 
conceptualization  provided  the  preliminary  theoretical  framework  required  to  develop  an  empirical 
model  of  the  relationship  between  aptitude  and  training  outcomes. 


"f 

COURSE  CONTENT 

+ 

* 

+  u 

' 

STUDENT  INPUTS 

EASE  OF  LEARNING 

TRAINING  OUTCOMES  COSTS 

+ 

1 

Figure  2.  Refined  Conceptual  Model 


Variable  Specification 

Following  development  of  the  general  conceptual  model,  the  next  step  In  this  effort  Involved 
Identifying  the  student  input  and  course  content  variables  capable  of  influencing  ease  of 
learning,  as  well  as  the  likely  relationships  among  these  variables.  Once  these  variables  had 
been  identified,  it  was  then  necessary  to  generate  measures  of  these  variables  and  ensure  the 
relevance  of  these  measures  to  training  performance.  In  order  to  obtain  this  Information,  It  was 
necessary  to  conduct  a  series  of  interviews  with  Air  Force  personnel  directly  Involved  In  the 
technical  training  process. 

A  set  of  preliminary  interviews  was  conducted  at  Headquarters  Air  Training  Command  (HQ  ATC) 
by  two  Advanced  Research  Resource  Organization  (ARRO)  staff  members  and  two  representatives  of 
AFHRL.  Interviewees  were  drawn  from  organizational  groups  charged  with  the  overall  control  and 
coordination  of  technical  training.  These  interviews  were  not  highly  structured  but  served  to 
provide  an  overview  of  the  technical  training  process.  Additionally,  at  this  time,  a  sample  of 
documentary  materials  were  obtained  including: 

1.  ATC  Forms  156,  Student  Record  of  Training, 

2.  plans  of  instruction  (POIs), 

3.  specialty  training  standards  (STSs),  and 

4.  course  charts  (CCs). 

The  information  obtained  in  the  preliminary  interview  was  then  used  to  design  a  more 
structured  interview  protocol  to  elicit  information  pertaining  to: 

1.  variables  indicative  )f  the  quality  of  student  training  performance; 

2.  student  input  variables  that  might  influence  training  performance  quality; 

3.  course  content  variables  that  might  influence  training  performance  quality; 

4.  probable  relationships  between  course  content  and  student  input  variables; 

5.  potential  measures  of  student  input,  course  content,  and  training  outcome  variables; 

6.  potential  biases  in  these  measures;  and 

7.  potential  sources  of  these  measures. 

An  example  of  the  interview  protocol  appears  in  Appendix  A.  Once  this  interview  protocol  had 
been  developed,  a  series  of  visits  were  then  made  to  four  ATC  technical  training  centers:  Lowry 
AFB,  Keesler  AFB,  Sheppard  AFB,  and  Chanute  AFB.  At  each  site,  interviews  were  conducted  with: 

1.  course  instructors, 

2.  instructor  supervisors, 

3.  training  managers, 

4.  instructional  systems  design  chiefs. 


5.  wing  educational  advisor, 


6.  measurement  section  personnel,  and 

7.  registrar. 

These  Individuals  were  chosen  so  that  several  different  perspectives  of  the  technical 
training  program  would  be  represented.  During  the  Interviews,  an  attempt  was  made  In  all  cases 
to  focus  discussion  on  those  questions  contained  In  the  general  Interview  protocol  most  relevant 
to  the  Interviewees'  areas  of  expertise. 

The  Interviews  were  conducted  by  two  ARRO  staff  members  and  two  AFHRL  representatives  during 
the  winter  of  1983.  One  ARRO  staff  member  was  responsible  for  guiding  the  discussion,  while  the 
other  was  responsible  for  generating  a  written  summary  of  the  discussion.  The  written  simnaries 
obtained  from  these  interviews  were  subsequently  reviewed  by  ARRO  staff  members  In  an  attempt  to 
Identify:  (a)  the  major  measurable  outcomes  of  resident  technical  training,  (b)  the  student 
Inputs  likely  to  Influence  outcomes  within  each  course  (and  across  courses),  and  (c)  the  course 
content  variables  likely  to  Influence  outcomes  within  each  course  (and  across  courses). 

The  review  also  focused  on  (a)  the  Interrelationships  among  student  Input  and  course  content 
variables  across  courses,  (b)  the  available  measures  of  these  constructs,  and  (c)  any  known 
biases  In  these  measures.  Once  this  Information  had  been  obtained.  It  was  used  to  specify  the 
variables  that  would  be  considered  for  Inclusion  in  the  model. 


Variables  and  Measures 

The  variables  considered  for  Inclusion  In  the  model  were  selected  on  the  basis  of  Information 
collected  during  the  Interviews.  Because  specification  of  variables  constitutes  a  critical  step 
in  the  development  of  a  valid  multivariate  model,  substantial  attention  is  given  to  this  topic 
below.  This  discussion  begins  with  an  examination  of  training  outcome  variables  and  then  turns 
to  the  student  Input  and  course  content  variables. 

Training  Outcomes.  To  understand  the  significant  outcomes  of  Initial-skills  training  In  ATC, 
It  Is  necessary  to  have  some  understanding  of  the  goals  of  the  training  effort.  In  the  Air 
Force,  Initial-skills  training  Is  not  Intended  to  produce  a  fully  qualified  journeyman  capable  of 
performing  all  tasks  within  a  specialty.  Rather,  students  are  trained  to  a  level  of  partial 
proficiency  in  the  tasks  they  are  most  likely  to  perform  upon  job  entry.  In  the  Air  Force,  this 
Is  called  "training  to  the  3-sklll  level."  The  assessment  of  trainee  performance  focuses  on  the 
acquisition  of  knowledge  and  performance  capabilities  associated  with  this  skill  level. 
Assessment  procedures  are  prescribed  in  ATC  Regulation  52-3,  Student  Measurement  Guidelines. 
This  document,  which  Is  presented  In  Appendix  B,  serves  to  specify  exactly  how  and  when  students 
should  be  evaluated.  Although  this  regulation  allows  for  some  local  variation  In  student 
evaluation  procedures.  Its  content  has  led  to  the  emergence  of  a  relatively  standard  set  of 
procedures. 

Typically,  a  technical  training  course  Is  divided  into  units  that  present  a  cohesive  body  of 
material  and  blocks  of  Instruction  that  are  composed  of  varying  numbers  of  units.  Within  each 
unit,  quizzes  or  progress  checks  (PC)  that  have  been  formally  specified  in  the  course  Plan  of 
Instruction  (POI)  are  administered.  Students  must  pass  each  PC  before  being  allowed  to  advance 
to  the  next  unit.  If  the  student  falls  a  PC,  the  instructor  is  expected  to  provide  Special 
Individual  Assistance  (SIA)  until  the  PC  has  been  passed,  although  SIA  may  be  provided  for  other 
reasons  as  well.  Once  the  student  has  passed  all  PCs  in  a  block,  an- end-of-block  written  test  is 
administered.  This  end-of-block  test  is  a  standard  paper- and-penc 11  academic  achievement  test. 
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The  student  must  obtain  a  minimum  score  of  70  on  the  block  test  to  advance  to  the  next  block  of 
Instruction.  If  a  falling  grade  Is  obtained,  the  student  may  be  required  to  repeat  the  block  of 
Instruction  before  retaking  the  test.  Failure  to  pass  on  the  second  attempt  usually  leads  to 
student  elimination. 

All  student  evaluations  are  recorded.  Progress  check  performance  Is  recorded  on  ATC  Form  98, 
Student  Progress  Checklist,  which  Is  destroyed  once  the  student  has  completed  the  course. 
End-of-block  tests  are  recorded  on  ATC  Form  156,  Student  Record  of  Training,  which  Is  maintained 
by  the  training  center  registrar's  office  as  a  permanent  record  of  an  Individual's  training 
performance.  An  example  of  ATC  Form  156  Is  provided  In  Appendix  C.  A  student's  final  grade  for 
the  course  Is  the  average  of  all  end-of-block  tests.  The  consensus  of  the  Interviewees  was  that 
the  average  final  course  grade  is  an  adequate,  if  not  excellent,  index  of  the  quality  of  student 
performance.  Therefore,  final  course  grade,  as  defined  by  average  performance  on  the  block 
tests,  was  used  to  provide  an  index  of  the  quality  of  student  training  performance  In  the  model. 

ATC  Form  156  also  provides  a  variety  of  additional  information  bearing  on  training 
performance.  One  such  item  Is  hours  of  SIA  time  provided  the  student.  Interviewees  Indicated 
that  SIA  time  was  accurately  recorded,  and  that  It  was  a  costly  and  Important  outcome.  Thus,  it 
seemed  prudent  to  consider  SIA  time  as  an  additional  training  outcome. 

ATC  Form  156  also  contains  information  concerning  two  other  outcomes  that  Interviewees 
considered  to  be  of  substantial  Importance.  All  negative  actions  concerning  a  student,  such  as 
elimination  from  training  or  scheduling  for  retraining,  must  be  preceded  by  a  counseling 
session.  In  these  counseling  sessions,  which  usually  last  about  an  hour,  student  performance  1« 
reviewed  by  an  instructor  supervisor  who  talks  to  the  student  and  determines  what  action  shoulo 
be  taken.  The  numbers  of  academic  and  nonacademic  counseling  sessions  are  recorded  on  the  ATC 
Form  156,  and  it  is  noted  whether  each  session  was  held  for  academic  or  nonacademic  reasons. 
Given  their  Importance,  it  seemed  necessary  to  quantify  counseling  sessions  as  a  training 
outcome.  This  was  done  by  determining  separately  the  total  number  of  academic  and  nonacademic 
counseling  sessions  a  student  received  during  the  course. 

Another  potential  outcome  of  training  (I.e.,  an  alternative  Index  of  student  performance)  Is 
retraining  time.  Retraining  occurs  when  a  student  is  required  to  repeat  one  or  more  blocks  of  a 
course  as  a  consequence  of  failing  an  end-of-block  test.  This  outcome  Is  significant  In  part 
because  retraining  is  remedial  action  taken  prior  to  elimination  for  academic  reasons,  and  In 
part  because  retraining  is  quite  expensive.  A  measure  of  retraining  time  may  be  obtained  from 
the  ATC  Form  156  by  subtracting  the  formal  course  length  from  the  individual's  total  hours  In 
training. 

As  implied  by  the  foregoing  discussion,  nearly  all  groups  Interviewed  Indicated  that  two  of 
the  most  important  training  outcomes  are  academic  and  nonacademic  elimination.  Both  academic  and 
nonacademic  eliminations  are  directly  recorded  on  ATC  Forms  156.  This  Information  was  used  to 
code  as  0's  students  who  graduated  and  to  code  as  I's  students  who  were  eliminated  from  training 
for  either  strictly  academic  or  strictly  nonacademic  reasons  (not  administrative  reasons  such  as 
death  or  lack  of  a  security  clearance).  Students  who  were  eliminated  presented  a  special  problem 
with  respect  to  the  measurement  of  SIA  time,  retraining  time,  academic  counseling,  nonacademic 
counseling,  and  the  quality  of  student  performance.  The  problem  was  to  estimate  expected  values 
for  these  variables  which  probably  would  have  been  observed  if  the  student  had  remained  In  the 
course  for  the  entire  training  period.  The  expected  value  of  quality  of  student  performance  was 
estimated  by  taking  the  average  of  all  block  test  grades  up  to  the  point  of  elimination.  For  SIA 
time,  retraining  time,  academic  counseling,  and  nonacademic  counseling,  the  expected  values  were 
estimated  by  calculating  the  rate  per  training  hour  up  to  the  point  of  elimination  and  Increasing 
the  observed  value  at  the  point  of  elimination  by  the  product  of  training  hours  not  completed  and 
the  rate  per  training  hour. 


Overall,  seven  training  outcomes  were  identified  as  being  significant  variables:  assessed 
quality  of  student  performance,  SIA  time,  retraining  time,  academic  counseling,  nonacademic 
counseling,  academic  attrition  (i.e.,  student  elimination),  and  nonacademic  attrition.  Measures 
of  all  these  variables  could  be  obtained  from  the  permanently  maintained  ATC  Form  156.  Although 
performance  on  PCs  would  seem  to  be  a  potentially  significant  outcome  variable,  this  information 
was  no  longer  available.  Based  upon  these  observations,  coupled  with  interviewees’  comments  that 
the  student  evaluation  data  were  accurately  recorded  and  represented  a  comprehensive  description 
of  training  performance,  it  seemed  reasonable  to  conclude  that  these  measures  would  provide  an 
adequate  basis  for  model  development. 

Student  Inputs.  Practically  all  interview  groups  agreed  that  a  student’s  aptitude 
significantly  influences  training  outcomes.  In  the  Air  Force,  at  least  three  measures  of 
aptitude  are  available:  Armed  Forces  Qualification  Test  (AFQT)  score,  ASVAB  subtest  scores,  and 
scores  on  the  ASVAB  aptitude  composites.  Given  current  Air  Force  classification  practices,  it 
was  decided  that  the  ASVAB  aptitude  composite  score  should  be  used  as  the  principal  Index  of 
aptitude.  This  decision  also  finds  some  support  in  the  fact  that  the  aptitude  composite  scores 
have  been  shown  to  be  reliable  and  valid  predictors  of  technical  training  performance  (Wllbourn 
et  al.,  1984).  In  the  present  effort,  although  it  was  possible  to  use  all  four  aptitude 
composite  scores  as  indicators  of  student  aptitudes,  it  was  deemed  more  appropriate  to  use  only 
the  aptitude  composite  score  that  applied  to  the  specialty  for  which  the  individual  was  being 
trained. 

Although  all  interviewees  considered  aptitude  to  be  of  great  Importance,  they  also  noted  that 
a  number  of  other  student  characteristics  Interacted  with  aptitude  in  determining  training 
performance.  For  instance,  trainers  noted  that  reading  ability  often  had  a  substantial  Impact  on 
training  performance,  due  to  the  reading  difficulty  of  the  technical  material  used  in  certain 
courses.  Currently,  two  measures  of  reading  ability  are  available  on  the  ATC  Form  156  which 
reflect  vocabulary  and  reading  comprehension  levels.  These  measures  are  derived  from  the  Air 
Force  Reading  Abilities  Test  (AFRAT),  which  is  currently  administered  during  basic  military 
training.  Alternate  forms  of  this  test  have  reliabilities  ranging  between  the  mid-. 80s  and 
low-. 90s  (Mathews  &  Roach,  1982).  Given  the  availability  of  these  measures  and  the  petceived 
effect  of  reading  ability  on  training  outcomes,  AFRAT  scores  were  Included  in  the  model  as  an 
average  of  the  vocabulary  and  reading  comprehension  scores. 

In  addition  to  aptitude  and  reading  ability,  course  supervisors  and  Instructors  emphasized 
the  Importance  of  educational  preparation  because  of  its  relation  to  study  habits  and  ability  to 
cope  with  an  academic  environment.  Currently,  the  Air  Force  does  not  use  direct  measures  of 
educational  preparation  in  the  classification  process;  however,  two  indirect  measures  were 
available.  The  first  of  these  was  the  educational  level  attained  by  the  individual  prior  to 
entering  the  Air  Force.  These  data  are  recorded  in  the  Processing  and  Classification  of 
Enlistees  (PACE)  personnel  data  file.  For  the  present  effort,  this  variable  was  scored  on  a 
5-point  continuum  such  that  non-high  school  graduates  received  a  score  of  1,  high  school 
graduates  received  a  score  of  2,  those  with  some  college  received  a  score  of  3,  college  graduates 
received  a  score  of  4,  and  those  having  post-graduate  work  received  a  score  of  5.  A  second 
measure  of  educational  preparation  was  also  obtained  from  the  PACE  file.  The  PACE  file  contains 
a  listing  of  some  42  different  high  school  courses  that  could  have  been  taken  by  the  individual 
prior  to  entering  the  Air  Force.  For  most  specialties,  the  Air  Force  lists  one  to  five  of  these 
courses  as  desirable  prerequisites  for  entry  into  an  AFS.  Thus,  an  additional  measure  of 
educational  preparation  could  be  obtained  by  summing  the  number  of  suggested  high  school  course 
prerequisites  that  the  individual  had  completed  prior  to  entering  the  Air  Force.  Although  the 
reliability  and  validity  of  these  particular  measures  were  not  established,  available  evidence 
concerning  such  background  data  measures  suggests  reliabilities  in  the  mid-. 90s  as  well  as 
substantial  predictive  power. 


The  high  school  course  records  contained  In  the  PACE  file  also  Bade  It  possible  to  assess 
another  student  Input  deemed  to  be  of  some  Importance  to  training  performance.  The  Interview 
groups  consistently  underscored  the  Importance  of  student  motivation  In  determining  training 
performance.  Although  neither  the  scope  of  the  present  effort  nor  the  current  Air  Force 
classification  procedures  would  permit  the  assessment  of  general  achievement  motivation.  It  was 
possible  to  build  a  measure  of  academic  achievement  motivation  from  the  available  background 
data.  This  was  done  by  having  five  psychologists  rate  the  overall  difficulty  of  each  of  the  42 
high  school  course  prerequisites  on  a  5-polnt  scale.  The  Interrater  reliability  of  these 
judgments  was  In  the  low-. 90s.  These  ratings  were  used  to  develop  an  Index  of  academic 
achievement  motivation  by  forming  a  weighted  sum  of  the  difficulty  of  the  high  school  courses 
taken  by  an  Individual.  Although  the  reliability  of  this  Index  Is  not  known,  given  the 
reliability  of  academic  history  data  and  the  reliability  of  the  difficulty  ratings.  It  appears 
that  the  reliability  of  the  index  would  be  fairly  high.  A  list  of  the  difficulty  ratings 
assigned  to  the  42  high  school  courses  is  provided  In  Appendix  D. 

During  the  interviews,  instructors  and  instructor  supervisors  often  noted  that  the  maturity 
of  the  student  was  of  some  importance  In  determining  training  outcomes.  At  least  two  Indicators 
of  maturity  were  available  for  use  In  the  present  effort.  The  first  of  these  was  age  at  the  time 
of  entry  into  training.  The  second  was  whether  the  student  was  a  new  enlistee,  as  opposed  to 
being  an  experienced  airman  retraining  for  a  new  specialty.  Although  the  latter  measure  appears 
to  be  a  somewhat  more  direct  index  of  vocational  maturity,  the  Infrequency  with  which  experienced 
airmen  are  retrained  into  totally  new  specialties  led  to  the  use  of  age  as  a  general  Index  of 
maturity.  Given  the  high  reliability  of  this  index,  the  limited  age  range  involved,  and  the 
control  for  education  level  and  aptitude  already  included  among  student  inputs.  Inclusion  of  this 
measure  seemed  well  justified. 

Students'  Interest  in  the  specialty  for  which  they  are  being  trained  was  also  cited  by  many 
Interview  groups  as  having  a  significant  influence  on  training  performance;  however,  few  Indices 
of  vocational  interest  are  available  In  the  Air  Force.  The  most  appropriate  of  these  Indices  is 
the  Air  Force  Vocational  Interest-Career  Examination  (Alley,  Wllbourn,  t  Berberlch,  1976). 
Unfortunately,  this  measure  is  not  yet  in  operational  use  and  so  could  not  be  employed  in  the 
present  effort.  Two  alternate  indices  of  Interest  could  also  be  identified,  however.  The  first 
of  these  was  a  measure  of  whether  the  trainee  had  or  had  not  been  guaranteed  training  In  a 
particular  specialty  as  part  of  his/her  enlistment  contract.  Trainees  having  guaranteed  training 
were  coded  as  2;  all  others  were  coded  as  1.  The  second  of  these,  also  obtained  from  the  PACE 
file,  was  a  combination  of  information  concerning  the  specialty  to  which  the  trainee  was  assigned 
and  a  rank-ordering  of  the  trainee's  five  specialty  preferences.  From  this  Information,  an 
Interest  index  was  derived  by  coding  as  a  5  trainees  who  got  their  first  choice;  by  coding  as  a  3 
trainees  who  got  their  second,  third,  fourth,  or  fifth  choice;  and  by  coding  as  a  1  trainees 
assigned  to  a  specialty  that  was  not  ranked.  Unfortunately,  when  selecting  specialties  for 
training,  enlistees  have  little  knowledge  concerning  the  content  and  nature  of  the  specialties. 
Thus,  these  two  vocational  Interest  measures  were  considered  to  be  of  limited  value,  and  were 
only  tentatively  considered  for  inclusion  in  the  model. 

At  least  three  other  classes  of  student  Input  variables  were  mentioned  as  being  related  to 
training  performance.  The  first  class  of  variables  Included  factors  such  as  resistance  to 
stress,  and  attention  to  detail.  Unfortunately,  measures  of  these  variables  are  not  routinely 
collected.  Moreover,  their  impact  on  performance  appears  to  be  AFS-specIf 1c,  In  the  sense  that 
resistance  to  stress  was  seen  as  being  particularly  Important  for  Air  Traffic  Controllers  but  not 
for  other  specialties  such  as  Personnel  Specialists.  This  specificity  argues  against  the  use  of 
such  variables  tn  formulating  a  general  model  of  technical  training;  and  so,  for  both  pragmatic 
and  theoretical  reasons,  variables  of  this  sort  were  not  included.  A  second  class  of  student 
Inputs  mentioned  as  being  related  to  training  performance  Involved  physical  ability.  However, 
the  specificity  of  occupational  physical  demands  and  their  limited  relevance  to  classroom 


performance  led  to  the  rejection  this  class  of  variables  despite  their  use  In  the  selection  and 
classification  process.  Finally,  demographic  variables  such  as  sex,  ethnicity,  and  socioeconomic 
status  were  not  considered  for  Inclusion  In  the  model  because  they  have  only  an  Indirect 
relationship  to  training  performance. 

Oesplte  the  exclusion  of  these  latter  variables.  It  appears  that  the  student  Input  variables 
specified  for  Inclusion  In  the  model  provide  a  reasonably  comprehensive  description  of  the 
relevant  student  attributes.  Measures  of  the  variables  selected  In  the  present  investigation 
were  obtained  from  the  PACE  file,  with  the  exception  of  AFRAT  scores,  which  were  obtained  from 
ATC  Form  156.  The  ready  availability  of  these  measures  and  their  general  significance  to 
training  appeared  to  provide  a  sound  practical  basis  for  model  development. 

Course  Content.  In  specifying  the  course  content  component  of  the  model,  a  variety  of  course 
variables  were  mentioned  by  Interviewees  as  being  of  some  Import.  One  of  these  variables  was 
course  length.  It  was  often  reported  that  time  constraints  were  an  Important  consideration  In 
determining  training  outcomes  and  that  given  sufficient  time,  anyone  could  be  trained.  Such 
comments  Indicated  that  course  length  should  be  Included  In  the  model.  This  variable  could  be 
measured  by  simply  obtaining  the  number  of  hours  of  classroom  Instruction  specified  In  the  course 
POI.  In  discussions  with  Instructors  and  course  supervisors,  it  was  also  noted  that  course 
training  programs  provided  either  a  full  8  hours  of  lecture  each  day  or  6  hours  of  lecture  plus  2 
hours  of  supervised  study.  Generally,  they  felt  that  what  was  called  the  "Improved''  or 
6-plus-2-hour  Instructional  day  contributed  to  better  performance  because  the  Information  load  on 
students  was  less,  thus  preventing  student  "burnout."  Thus,  the  length  of  the  Instructional  day 
was  measured  by  coding  an  8-hour  day  as  a  1  and  a  6-p1us-2-hour  day  as  a  0.  In  the  case  of  both 
Instructional  day  length  and  course  length,  these  measures  can  be  assumed  to  be  perfectly 
reliable  due  to  the  highly  structured  and  tightly  scheduled  nature  of  resident  technical  training. 

Another  topic  that  received  a  great  deal  of  attention  In  the  Interviews  concerned  the  quality 
and  experience  of  Instructors.  Course  supervisors  and  branch  chiefs  pointed  out  that  Instructor 
experience  In  the  classroom  often  seemed  related  to  the  quality  of  student  performance,  as  did 
the  overall  quality  of  the  Instruction  provided  to  students. 

Instructor  quality  is  assessed  through  supervisory  ratings  and  Is  recorded  on  ATC  Form  281, 

Instructor  Evaluation  Checklist.  This  form  Is  a  behavioral  checklist  containing  19 

nonadmlnlstrative  Items  which  had  an  Interrater  reliability  (Rklt)  of  .83  for  a  sample  of  100 
Instructors  drawn  from  48  different  courses.  These  Instructor  evaluations  are  conducted  at 
specified  intervals,  depending  on  the  particular  Instructor's  experience.  Performance  on  each 
Item  Is  graded  as  good,  satisfactory,  or  poor.  These  evaluations  are  retained  as  permanent 
records  and  could  be  easily  obtained.  Although  some  concern  with  their  accuracy  was  voiced 
during  the  Interviews,  It  was  felt  that  they  provided  a  useful  Index  of  the  quality  of  the 

Instruction  being  provided  In  a  course.  Ratings  on  each  Item  were  quantified  by  assigning  good 
ratings  a  score  of  3,  satisfactory  ratings  a  score  of  2,  and  poor  ratings  a  score  of  1.  The 
quality  of  individual  Instructors  was  calculated  as  an  average  of  the  Item  ratings,  while  quality 
of  Instruction  at  the  course  level  was  determined  by  averaging  the  ratings  received  by  all 

Instructors  assigned  to  the  course.  Instructor  experience  presented  a  somewhat  less  complex 
variable  in  terms  of  measurement  procedures  since  interviewees  Indicated  that  the  number  of 
months  of  Instructional  experience  possessed  by  each  Instructor  could  be  readily  provided  by 
course  supervisors.  Thus,  a  reliable  Index  of  course-level  Instructor  experience  could  be 
obtained  simply  by  calculating  the  average  months  of  experience  of  all  Instructors  assigned  to 
the  course. 

For  any  given  course,  the  ratio  of  the  number  of  students  to  the  number  of  faculty  members 
was  also  considered  as  a  variable  contributing  to  student  performance  because  the  fewer  students 
per  Instructor,  the  more  Instructional  assistance  time  was  potentially  available  to  any  one 


student.  It  was  noted  that  student-faculty  ratio  is  a  fixed  course  variable  specified  in  the 
training  plan  (TP)  developed  for  each  course.  Student-faculty  ratio  listed  in  the  training  plan 
was  employed  as  a  fully  reliable  measure  of  this  course  content  variable. 

The  amount  of  feedback  and  practice  given  to  students  has  long  been  considered  to  have  an 
effect  on  training  performance.  This  point  was  underscored  by  the  training  personnel 
Interviewed.  They  pointed  out  that  student  assessment  by  means  of  PCs,  and  the  resulting  rapid 
feedback,  improved  later  performance  on  the  end-of-block  written  tests.  The  interviewees  also 
Indicated  that  formal  feedback  in  terms  of  the  progress  checks  and  block  tests  was  specified  in 
the  POI,  and  that  this  student  evaluation  schedule  was  rigidly  adhered  to.  Thus,  it  was  possible 
to  generate  a  reliable  measure  of  the  amount  of  formal  feedback  in  a  course  by  sunning  the  number 
of  progress  checks  and  block  tests  specified  In  the  POi.  However,  it  seemed  necessary  to  control 
for  course  length  in  the  development  of  this  measure.  Hence,  this  sum  was  divided  by  the  total 
number  of  instructional  hours  to  obtain  an  index  of  the  amount  of  feedback  per  unit  time. 

To  address  the  issue  of  practice.  It  was  necessary  to  establish  the  average  amount  of 
training  time  students  were  given  on  a  unified  body  of  material.  This  was  accomplished  by 
determining  the  total  number  of  units  in  a  course  and  then  dividing  the  total  number  of 
instructional  hours  by  the  number  of  units  to  determine  the  average  amount  of  time  on  each  unit 
of  Instruction.  Again,  this  information  was  obtained  from  the  POI.  The  documentary  nature  of 
these  data  would  indicate  perfect  reliability  for  this  measure,  at  least  In  the  limited  sense 
defined  here.  It  should  also  be  noted  that  since  student  evaluation  in  technical  training  Is 
closely  linked  to  the  completion  of  instructional  units,  it  was  assumed  that  there  would  be  a 
strong  negative  relationship  between  practice  and  feedback. 

In  the  course  of  the  Interviews,  instructors  and  Instructor  supervisors  often  stated  that 
performance  could  be  improved  by  the  use  of  instructional  aids  and  hands-on  practice.  It  was 
noted  that  the  number  of  aids,  as  defined  by  job  task  simulations,  mockups,  and  direct  use  of  job 
materials,  was  formally  specified  in  the  course  POI  by  unit  of  instruction.  Thus,  a  direct  and 
reliable  Indicator  of  the  extent,  to  which  instructional  aids  were  used  in  a  course  could  be 
obtained  by  counting  the  mrntber  of  times  mockups,  job  task  simulations,  or  direct  manipulations 
of  job  materials  were  required.  Similarly,  the  POI  specified  the  number  of  hands-on  hours  of 
instruction  by  course  unit  and  so  a  reliable  measure  of  this  variable  could  be  generated  by 
sunning  the  number  of  hands-on  hours  across  Instructional  units.  However,  both  the  number  of 
aids  and  the  amount  of  hands-on  training  are  obviously  dependent  on  course  length.  To  control 
for  this  biasing  influence,  both  the  total  number  of  aids  and  the  number  of  hands-on  hours  were 
divided  by  course  length  to  yield  indices  of  aids  and  hands-on  practice  per  unit  time. 

When  instructional  systems  design  personnel,  training  managers,  and  educational  advisors  were 
Interviewed,  it  was  often  pointed  out  that  AFS  manning  requirements  constituted  a  significant 
Influence  on  the  design  of  the  associated  course  via  a  variety  of  direct  and  Indirect  routes, 
ranging  from  their  Influence  on  the  aptitude  of  trainees  to  the  frequency  of  feedback.  Two 
indices  of  manning  requirements  were  obtained.  First,  an  index  could  be  found  In  the  number  of 
students  trained  In  a  calendar  year.  Another  index  of  manning  requirements  was  found  In  the 
availability  of  a  monetary  reenlistment  bonus  for  the  associated  job  specialty.  Such  bonuses  are 
provided  when  there  is  a  need  for  manpower  in  a  particular  job  specialty.  Specialties  provided 
reenlistment  bonuses  were  coded  as  1  and  specialties  not  provided  bonuses  were  coded  as  0  In 
measuring  this  variable.  The  number  of  students  trained  per  year  was  obtained  directly  from  the 
course  training  plan. 

In  addition  to  manning  needs,  the  difficulty  of  tasks  performed  in  a  job  specialty  was 
frequently  mentioned  by  instructional  systems  design  personnel  and  training  managers  as  having  a 
significant  influence  on  the  design  of  courses  and  on  student  performance.  As  a  result,  it 
seemed  prudent  to  include  an  index  of  the  difficulty  of  job  tasks.  Given  the  objectives  of  the 


present  effort.  It  seemed  appropriate  to  use  the  occupational  learning  difficulty  Indices 
resulting  from  AFHRL  occupational  research  as  a  measure  of  the  variable.  Information  pertaining 
to  the  validity  of  this  construct  Is  provided  elsewhere  (Burtch  et  al.,  1982;  Weeks,  Mumford,  & 
Harding,  1985). 

As  well  as  the  difficulty  of  the  tasks  being  trained,  nearly  all  Interview  groups  mentioned  a 
variety  of  other  factors  that  Influenced  the  difficulty  level  of  technical  training  courses.  One 
variable  frequently  mentioned  was  the  degree  of  abstractness  of  the  course  training  material. 
Unfortunately,  no  measure  of  this  construct  was  readily  available;  however,  this  concept  appeared 
to  be  of  sufflcent  importance  to  warrant  its  Inclusion  In  the  model.  Consequently,  It  was 
decided  to  obtain  ratings  of  the  degree  of  abstractness  of  course  material.  This  was  done  by 
having  five  psychologists  familiarize  themselves  with  the  plan  of  Instruction,  training  plan,  and 
a  sample  of  course  material  for  each  course.  They  then  used  a  5-point  rating  scale  to  judge  each 
course  In  terms  of  the  degree  to  which  the  mastery  of  abstract  principles  and  concepts  was 
emphasized.  An  average  of  these  ratings  was  then  used  to  represent  the  abstractness  of  each 
course  as  a  whole.  For  a  sample  of  48  courses,  an  interrater  reliability  (Rkk)  of  .88  was 
obtained  for  this  Index.  In  addition,  the  abstractness  values  assigned  various  courses  tended  to 
conform  to  generally  accepted  notions  concerning  course  difficulty.  For  example,  the  precision 
measuring  equipment  course  received  a  high  rating,  whereas  the  security  and  law  enforcement 
courses  were  rated  lower. 

Another  variable  that  was  viewed  by  instructional  systems  design  personnel  and  instructors  as 
being  critical  to  training  outcomes  was  the  reading  difficulty  of  course  materials.  This  was 
especially  true  in  the  more  demanding  training  programs.  Consequent ly,  an  attempt  was  made  to 
assess  the  reading  difficulty  of  course  materials.  For  each  course,  one  paragraph  from  each  of 
five  different  course  readings  was  randomly  selected.  Subsequently,  the  reading  difficulty  of 
each  paragraph  was  assessed  using  an  algorithm  developed  by  Taylor,  Sticht,  Fox,  and  Ford 
(1973).  This  algorithm  has  been  shown  to  exhibit  reliabilities  in  the  .90s.  To  compute  the 
overall  Index  of  reading  difficulty  for  each  course,  the  average  reading  difficulty  level  of  all 
five  of  the  selected  paragraphs  was  derived. 

Another  potential  source  of  course  difficulty  Identified  In  the  Interviews  pertained  to  the 
diversity  of  course  materials.  It  was  pointed  out  that  courses  were  more  difficult  when  students 
were  required  to  master  a  variety  of  material.  This  variable  proved  to  be  measurable  on  the 
basis  of  available  course  Information.  Because  the  units  of  Instruction  within  each  course 
reflect  a  unified  body  of  material,  a  fully  reliable  Index  of  diversity  was  derived  simply  by 
counting  the  number  of  units  specified  In  the  course  P01. 

One  final  course  content  variable  that  emerged  in  discussions  with  training  personnel  was  the 
expected  student  attrition  rate  for  the  course.  The  expected  attrition  rate  not  only  Influences 
the  number  of  students  assigned  to  a  course,  but  it  was  believed  that  It  also  influences  a 
variety  of  decisions  In  course  design,  as  well  as  training  outcomes  such  as  Individual  student 
attrition  decisions.  Thus,  there  seemed  to  be  a  need  to  obtain  an  adequate  measure  of  this 
variable.  This  presented  little  difficulty  because  expected  attrition  rates  arm  specified  In  the 
course  training  plan.  Consequently,  this  Information  could  be  employed  to  provide  a  highly 
reliable  measure  of  this  construct. 

For  the  most  part,  the  course  content  variables  outlined  previously  were  those  that  the 
Interview  groups  considered  to  have  the  greatest  Influence  on  student  performance  in 
Initial-skills  training.  However,  at  least  four  course  content  variables  mentioned  in  the 
Interviews  were  not  included  in  the  modeling  effort;  these  were  student-equipment  ratio, 
patterning  of  the  academic  day  between  lecture  and  discussion,  swing  shift  versus  day  shift,  and 
volunteer  versus  nonvolunteer  Instructors.  The  latter  two  variables  were  rejected  because  they 


could  not  be  quantified  without  extraordinary  efforts.  The  other  variables  did  not  appear  to 
have  sufficient  impact,  on  the  basis  of  the  interview  data,  to  warrant  their  inclusion  in  the 
model. 


The  preceding  discussion  has  outlined  the  course  content,  student  input,  and  training  outcome 
variables  that  were  considered  for  inclusion  in  the  modeling  effort,  along  with  potential 
measures  of  these  variables.  An  overview  of  all  variables  and  associated  measures  is  presented 
in  Table  1.  The  comprehensiveness  of  the  interviews  suggests  that,  for  the  most  part,  all  major 
variables  were  included. 


Sample  Selection  and  Data  Collection 

Construction  of  a  general  model  applicable  to  a  variety  of  courses  requires  that  a 
representative  sample  of  technical  training  courses  be  examined  for  the  purpose  of  model 
development  and  testing.  Accordingly,  it  was  decided  that  data  would  be  obtained  from  a  sample 
of  50  initial-skills  courses.  Forty  of  these  courses  were  to  be  used  in  model  development  while 
the  remaining  10  courses  would  be  used  for  cross-validation.  The  courses  in  model  development 
and  cross-validation  were  selected  by  AFHRL  in  consultation  with  representatives  of  ATC  and  the 
Air  Force  Military  Personnel  Center.  Selection  of  courses  was  based  on  several  criteria  such  as: 

1.  The  selected  courses  should  be  representative  of  all  ATC  Initial-skills  courses. 

2.  The  selector  aptitude  composites  for  the  courses  should  represent  all  four  aptitude 
areas,  but  courses  having  dual  selector  composites  should  be  excluded. 

3.  There  should  be  adequate  variance  in  the  minimum  aptitude  requirements  for  the  courses 
selected. 


4.  There  should  be  adequate  variance  in  the  content  of  the  courses. 

5.  Different  ATC  technical  training  centers  should  be  represented. 

6.  Relatively  high-cost  courses  should  be  included,  as  well  as  courses  with  high  student 

flow. 

7.  Courses  having  classified  training  documents  (e.g.,  POIs)  should  be  excluded. 

8.  Recently  modified  courses  should  be  excluded. 

9.  Courses  having  computer-assisted  instruction  (CAI)  should  be  represented. 

10.  Courses  associated  with  sortie-generating  specialties  should  be  represented. 

Once  the  sample  of  courses  had  been  Identified,  personnel  and  training  data  required  for 
model  development  and  cross-validation  were  collected.  The  student  sampling  strategy  was 
relatively  straightforward.  For  each  course,  existing  data  bases  were  employed  to  identify  a 
minimum  of  50  individuals  who  had  been  admitted  to  one  of  the  five  most  recent  training  classes. 
Subsequently,  all  other  trainees  who  had  entered  the  course  during  the  previous  6  months  were 
identified.  AFHRL  supplied  this  information  along  with  all  relevant  personnel  information  drawn 
from  the  PACE  file.  Visits  were  then  made  to  Sheppard  AFB,  Chanute  AFB,  Lowry  AFB,  Keesler  AFB, 
and  Lackland  AFB,  where  ARRO  staff  members  reproduced  the  training  records  of  all  students 
selected  for  inclusion  in  the  study.  At  the  same  time,  for  each  course,  the  following  training 
documents  were  obtained: 


Table  1.  Specification  of  Measures 


Measure 

Source 

Training  Outcome  Variables 

Quality  of  Student 

Average  of  end-of-block  tests 

Student  Record  of  Training, 

ATC 

156 

Performance 

SIA  Time 

Hours  of  special  individual  assistance  (SIA) 

Student  Record  of  Training, 

ATC 

156 

Academic  Counseling 

Number  of  academic  counseling  sessions 

Student  Record  of  Training, 

ATC 

156 

Nonacademic  Counseling 

Number  of  nonacademic  counseling  sessions 

Student  Record  of  Training, 

ATC 

156 

Retraining  Time 

Number  of  hours  of  retraining 

Student  Record  of  Training, 

ATC 

156 

Academic  Attrition 

Percentage  of  student  academic  eliminations 

Student  Record  of  Training, 

ATC 

156 

Nonacademic  Attrition 

Percentage  of  student  nonacademic  eliminations 

Student  Record  of  Training, 

ATC 

156 

Student  Input  Variables 

Aptitude 

Scores  on  selector  aptitude  composite  of 

Personnel  Data  Files 

Armed  Services  Vocational  Aptitude  Battery 

Reading  Level 

Average  score  on  the  Air  Force  Reading 

Student  Record  of  Training, 

ATC 

156 

Abilities  Test 

Academic  Motivation 

Number  of  difficult  high  school  courses  taken 

Personnel  Data  Files 

Simple  Interest 

Received  guaranteed  specialty 

Personnel  Data  Files 

Preference  Interest 

Received  preferred  specialty 

Personnel  Data  Files 

Educational  Level 

Highest  educational  level 

Personnel  Data  Files 

Educational  Preparation 

Recommended  high  school  course  prerequisites 

Personnel  Data  Files 

taken 

Age 

Years  from  birth 

Personnel  Data  Files 

Course  Content  Variables 

Course  Length 

Number  of  total  instructional  hours 

Plan  of  Instruction  (POI) 

Day  Length 

Length  of  instructional  day  (1  equals  8- 

Training  Plan 

hour  day;  0  otherwise) 

Student-Faculty  Ratio 

Number  of  students  per  Instructor 

Training  Plan 

Instructor  Experience 

Average  months  of  instructor  experience 

Plan  of  Instruction  (POI) 

Instructor  Quality 

Average  instructor  performance  evaluations 

Course  Supervisors 

Aids  in  Use 

Number  of  instructional  aids  divided  by 

Plan  of  Instruction  (POI) 

course  length 

Hands-On  Instruction 

Hours  of  hands-on  instruction  divided  by 

Plan  of  Instruction  (POI) 

course  length 

Amount  of  Feedback 

Number  of  evaluations  divided  by  course 

Plan  of  Instruction  (POI) 

length 

Amount  of  Practice 

Course  length  divided  by  number  of  units 

Plan  of  Instruction  (POI) 

Reenlistment  Bonus 

1  If  selective  reenlistment  bonus  offered; 

AFHRL 

otherwise  0 

Yearly  Student  Flow 

Number  of  students  to  be  trained  yearly 

Training  Plan 

Occupational  Difficulty 

Overall  learning  difficulty  of 

occupational  tasks 

AFHRL 

Reading  Difficulty 

Average  reading  grade  level  of  course 

ARRO  Staff 

materials 

Abstract  Knowledge 

Average  rating  of  abstract  knowledge 

ARRO  Staff 

requirements 

Expected  Attrition 

Expected  proportion  of  students  eliminated 

Training  Plan 

Diversity 

Number  of  units  in  course 

Plan  of  Instruction  (POI) 

1.  plan  of  instruction  (POI) 


2.  course  chart  (CC), 

3.  training  plan  (TP), 

4.  specialty  training  standard  (STS), 

5.  a  sample  of  training  materials, 

6.  number  of  months  of  instructional  experience  of  instructors  assigned  to  the  course,  and 

7.  the  two  most  recent  instructor  evaluation  checklists  (ATC  Form  281)  for  all  Instructors 
assigned  to  the  course. 

An  overview  of  the  data  collected  may  be  found  in  Table  2.  As  may  be  seen,  a  complete  set  of 
usable  data  was  obtained  for  48  of  the  50  courses  identified.  The  courses  for  which  complete 
data  could  not  be  obtained  were  associated  with  AFS  42334,  Aircraft  Pneudraulic  Systems  Mechanic, 
and  AFS  32637,  Integrated  Avionics  Attack  Control  Systems  Specialist.  This  resulted  In  the  loss 
of  one  course  from  both  the  cross-validation  and  model-development  samples.  As  a  result,  all 
personnel  and  training  data  collected  for  these  courses  were  eliminated  from  the  analyses. 
Nevertheless,  as  is  shown  in  Table  2,  the  remaining  model-development  and  cross-validation 
courses  were  quite  diverse,  with  titles  ranging  from  Carpentry  Specialist  and  Financial  Service 
Specialist  to  Precision  Measuring  Equipment  Specialist  and  Jet  Engine  Mechanic.  In  all,  eight 
courses  were  drawn  from  the  technical  training  center  (TTC)  at  Chanute  AFB,  14  from  the  TTC  at 
Sheppard  AFB,  10  from  the  TTC  at  Lowry  AFB,  14  from  the  TCC  at  Keesler  AFB,  and  two  from  the 
Military  Training  Center  at  Lackland  AFB.  Because  five  of  the  six  major  ATC  training  centers 
were  represented  in  the  present  effort,  representation  was  considered  adequate.  Moreover,  the 
selector  aptitude  composites  associated  with  the  selected  courses  represent  all  aptitude  areas 
and  the  full  range  of  minimum  aptitude  requirements.  In  sum,  these  observations  suggest  that  the 
courses  identified  for  model  development  and  cross-validation  provide  a  representative  sample  of 
all  Air  Force  initial-skills  courses. 

For  all  courses  combined,  the  student  sample  provided  a  total  of  5,981  subjects  for  model 
development  and  cross-validation.  All  students  in  the  sample  entered  training  in  1983  or  1984 
and  had  completed  training  by  August  1984.  The  vast  majority  of  the  students  were  males; 
however,  female  students  were  included  in  proportion  to  their  overall  presence  in  the  Air  Force. 
Within  the  total  sample,  data  for  5,091  students  were  to  be  used  for  model  development,  and  data 
for  890  students  were  to  be  used  In  the  cross-validation.  There  were  no  marked  differences  in 
the  demographic  characteristics  of  the  student  samples  used  for  the  model  development  and 
cross-validation. 


Preliminary  Analyses 

Once  the  data  had  been  collected  and  coded,  preliminary  data  analyses  were  conducted.  In 
these  analyses,  it  was  assumed  that  all  course  content  variables  could  be  applied  to  all  students 
assigned  to  each  course.  Further,  it  was  assumed  that  students  with  missing  data  for  some 
variable  would  be  omitted  from  only  those  analyses  which  Involved  that  variable.  Given  these 
assumptions,  the  overall  mean  and  standard  deviation  for  each  course  content,  student  input,  and 
training  outcome  variable  were  obtained  in  the  total  sample,  the  model-development  sample,  and 
the  cross-validation  sample.  Subsequently,  the  mean  and  standard  deviation  of  each  student 
input,  course  content,  and  training  outcome  variable  were  obtained  for  each  course. 
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After  these  preliminary  descriptive  analyses  had  been  conducted,  an  attempt  was  made  to 
establish  the  simple  bivariate  relationships  among  all  variables.  This  was  accomplished  by 
intercorrelating  scores  on  all  of  the  variables  being  considered  for  inclusion  in  the  model  in 
the  total  sample,  and  then  obtaining  separate  estimates  of  these  relationships  within  the 
model-development  and  cross-validation  samples.  This  basic  descriptive  information  provided  a 
framework  for  reviewing  the  initial  conceptual  model,  and  reevaluating  initial  conceptualizations 
of  the  relationships  among  the  variables.  This  revision  and  refinement  of  the  initial 
hypothetical  model  was  a  highly  intuitive  process  carried  out  in  a  series  of  discussions  with 
knowledgeable  Air  Force  personnel. 


;  Primary  Analyses 

J  Once  an  acceptable  revision  of  the  initial  hypothetical  model  had  been  formulated,  model 

!  development  was  undertaken.  This  was  a  relatively  straightforward  process  in  which  the  LISREL  V 

program  for  the  estimation  of  linear  structural  models  was  applied  to  the  correlations  obtained 
I  within  the  model-development  sample  (Joreskog  &  Sorbom,  1980).  All  analyses  were  carried  out 

J  using  an  unweighted  least  squares  procedure.  Application  of  this  strategy  offered  certain 

I  advantages  in  terms  of  cost  and  the  robustness  of  parameter  estimates,  although  it  did  prohibit 

j  obtaining  estimates  of  the  standard  error  of  the  path  coefficients. 

The  LISREL  V  program  operates  by  taking  a  series  of  specified  causal  relationships  among  the 
variables  and  generating  parameter  estimates  that  will  maximize  the  amount  of  covariation  in  the 
observed  data  which  can  be  accounted  for  by  the  hypothetical  model.  As  well  as  allowing  the  user 
to  specify  causal  paths  between  exogenous  (i.e.,  independent)  variables  and  endogenous  (i.e., 

|  dependent)  variables,  LISREL  V  allows  specification  of  causal  paths  among  endogenous  variables, 

j  In  estimating  these  parameters,  the  LISREL  V  program  can  take  into  account  correlations  among 

,  observed  or  latent  exogenous  variables  and  endogenous  variables,  as  well  as  correlated  error 

.  terms  among  exogenous  variables  and  correlated  error  terms  among  endogenous  variables.  In  all 

|  analyses,  correlation  matrices  were  assumed  to  provide  legitmate  estimates  to  be  included  in  the 

•  model. 

Once  these  decisions  had  been  made,  it  was  possible  to  run  the  model  through  the  LISREL  V 
program.  Initially,  the  goodness-of-f it  test  and  the  amount  of  residual  variance  unaccounted  for 
,  by  the  model  were  examined.  The  multiple  correlation  coefficients  (Rs)  for  the  prediction  of  all 

training  outcomes  given  the  hypothesized  causal  relationships  specified  by  the  model  were 
obtained,  along  with  regression  coefficients  and  standardized  path  coefficients  specified  by  the 
causal  relationships  among  the  exogenous  and  endogenous  variables  and  among  the  endogenous 
variables  alone.  Estimates  of  the  total  effect  of  each  variable  on  all  relevant  dependent 
variables  were  also  generated  along  with  the  requisite  correlation  matrices.  It  should  be  noted 
!  that  these  regression  weights  and  path  coefficients  are  not  identical  to  Rs  but  rather,  represent 

|  weights  for  prediction. 

* 

|  Once  this  model  had  been  generated  and  the  relevant  data  examined,  an  attempt  was  made  to 

|  evaluate  the  predictive  efficiency  of  the  model.  The  regression  coefficients  specified  in  the 

[  foregoing  analyses  were  used  to  predict  training  outcomes  in  the  nine  cross-validation  courses, 

J  given  knowledge  of  the  course  content  variables  for  each  course  and  average  values  for  student 

i  input  variables.  The  agreement  between  predicted  training  outcomes  and  the  observed  training 

j  outcomes  for  each  of  the  nine  courses  then  served  as  the  basis  for  evaluating  the  predictive 

'  utility  of  the  model.  On  the  whole,  the  analyses  described  above  seemed  adequate  for  the 

|  development  of  an  empirical  model  for  use  in  predicting  the  impact  of  aptitude  requirement 

\  adjustments  on  Air  Force  initial-skills  training. 


III.  RESULTS 


Descriptive  Analyses 

Table  3  presents  means  and  standard  deviations  of  scores  on  all  of  the  student  Input,  course 
content,  and  training  outcome  variables  for  the  total  sample,  the  model-development  sample,  and 
the  cross-validation  sample.  These  data  were  derived  by  aggregating  over  students  or  courses 
within  each  sample.  The  actual  value  for  each  student  input,  course  content,  and  training 
outcome  variable  for  each  technical  training  course  are  provided  in  Appendices  E,  F,  and  G.  With 
regard  to  student  inputs,  it  should  be  noted  that  ASVAB  scores  for  all  students  are  based  on  1944 
norms.  This  fact  is  important  for  score  interpretation  because  in  October  1984,  soon  after  data 
had  been  collected  for  this  effort,  a  new  set  of  norms  based  on  a  1980  youth  population  began  to 
be  used  for  the  derivation  of  ASVAB  scores  (Ree,  Valentine,  &  Earles,  1985).  Comparisons  between 
the  mean  scores  on  the  ASVAB  aptitude  composites  for  each  sample  and  historical  ASVA8  data 
indicated  that  sample  means  are  typical  of  those  found  in  Air  Force  enlistee  populations. 
Examination  of  Table  3  indicates  that  most  trainees  were  approximately  20  years  old  and  had 
completed  high  school.  Additionally,  they  had  taken  one  or  two  courses  in  high  school  that  were 
considered  to  be  desirable  prerequisites  for  the  specialties  to  which  they  were  assigned.  The 
interest  measures  indicate  that  roughly  half  of  the  trainees  were  guaranteed  a  particular  Job 
specialty  and  that  most  trainees  received  one  of  their  preferred  specialties.  Typically,  these 
trainees  were  able  to  read  at  an  eleventh  grade  level,  and  had  evidenced  some  academic 
achievement  motivation  with  respect  to  the  courses  they  took  in  high  school. 

On  the  whole,  the  means  and  the  standard  deviations  of  the  various  student  input  variables 
were  remarkably  similar  for  the  model-development  and  cross-validation  samples.  However,  It 
appears  that  the  cross-validation  sample  consisted  of  students  of  somewhat  higher  quality.  This 
was  evidenced  by  consistently  higher  average  scores  on  the  ASVAB  aptitude  composites  In  the 
cross-validation  sample.  This  result  appears  to  indicate  that  the  cross-validation  sample  was 
composed  of  courses  associated  with  higher  aptitude  requirement  minimums. 

In  reviewing  the  results  obtained  for  the  course  content  variables,  a  number  of  salient 
points  should  be  noted.  First,  the  resident  technical  training  courses  sampled  were  about  400 
hours  in  length  and,  on  the  average,  were  divided  into  49  units  of  Instruction  (diversity). 
Roughly  half  of  the  courses  operated  on  an  8-hour  instructional  day,  whereas  the  other  half 
operated  on  a  6-plus-2-hour  instructional  day.  Generally,  one  instructor  was  available  for  every 
nine  students.  Instructors  tended  to  have  roughly  2  1/2  years  of  training  experience,  and  the 
quality  of  the  instruction  they  provided  was  judged  by  instructor  supervisors  as  being  between 
good  and  satisfactory.  Slightly  less  than  half  of  the  the  total  training  time  was  devoted  to 
hands-on  instruction,  while  instructional  aids  were  used  on  at  least  a  daily  basis  and  feedback 
in  one  form  or  another  was  provided  once  every  3  hours.  On  the  average,  training  programs 
devoted  8.5  hours  to  each  unit  of  instruction,  and  operated  under  an  expected  attrition  rate  of 
9%.  Of  the  courses  selected  for  study,  slightly  less  than  half  were  associated  with  specialties 
having  selective  reenlistment  bonuses.  On  the  whole,  most  courses  made  only  moderate  demands 
with  regard  to  the  mastery  of  abstract  principles  although  there  were  substantial  differences  In 
this  variable  across  courses.  The  average  reading  grade  level  of  training  materials  was  10.4, 
indicating  that  the  material  was  quite  appropriate  for  use  with  students  having  an  11.4  reading 
grade  level.  However,  it  should  be  recognized  that  this  conclusion  pertains  only  to  training 
materials  produced  by  ATC.  Training  materials  produced  by  private  vendors  and  technical  orders 
were  not  included  in  the  analysis.  The  average  learning  difficulty  of  the  tasks  performed  in 
specialties  associated  with  the  courses  sampled  was  typical  of  that  found  for  most  AFSs.  It 
should  be  noted  that  the  average  yearly  student  flow  for  the  courses  was  quite  high  In  the  total 
and  model-development  samples  relative  to  the  cross-validation  sample.  This  result  Is  due  to  the 
inclusion  of  a  few  high-flow  courses,  such  as  Security  Specialist  (AFS  81130),  In  the 


Table  3.  Means  and  Standard  Deviations  of  Variables 


Total  sample 
(N  -  5.981) 

Model -deve 1 opment 
sample 
(N  -  5.091) 

Cross-validation 
sample 
(N  -  890) 

Student  input  variables9 

X 

SO 

X 

SD 

X 

SD 

Mechanical 

55.13 

22.40 

54.40 

22.30 

59.30 

22.60 

Administrative 

63.16 

19.50 

62.70 

19.50 

65.80 

19.70 

General 

65.28 

16.90 

64.70 

16.80 

68.60 

17.30 

Electronics 

63.46 

18.50 

62.70 

18.50 

67.80 

18.00 

Reading  Level 

11.43 

1.00 

11.40 

1.00 

11.60 

1.00 

Academic  Motivation 

38.94 

13.40 

38.40 

13.30 

42.00 

13.50 

Simple  Interest 

1.41 

.49 

1.40 

.49 

1.50 

.50 

Preference  Interest 

4.42 

• 

00 

c* 

4.40 

.87 

4.50 

.82 

Educational  Level 

2.12 

.45 

2.10 

.42 

2.20 

.56 

Educational  Preparation 

1.65 

.94 

1.60 

.87 

1.90 

1.10 

Age 

20.07 

2.20 

20.00 

2.10 

20.50 

2.30 

Course  content  variables** 


Course  Length 

420. 73 

309.30 

386.30 

302.20 

569.90 

297.70 

Day  Length 

.49 

.49 

.46 

.49 

.62 

.48 

Student-Faculty  Ratio 

9.03 

4.80 

9.20 

4.90 

8.30 

3.90 

Instructor  Experience 

33.43 

14.70 

29.70 

12.90 

49.60 

13.60 

Instructor  Quality 

2.48 

.16 

2.50 

.16 

2.40 

.13 

Aids  In  Use 

.27 

.10 

.27 

.10 

.28 

.13 

Hands-On  Instruction 

.40 

.13 

.41 

.13 

.37 

.10 

Amount  of  Feedback 

.34 

.12 

.35 

.13 

.28 

.06 

Amount  of  Practice 

8.49 

3.10 

8.60 

3.00 

8.  CO 

3.80 

Reenlistment  Bonus 

.46 

.49 

.42 

.49 

.61 

.48 

Yearly  Student  Flow 

1852.33 

2662.50 

2204.90 

2782.00 

324.50 

118.00 

Occupational  Difficulty 

99.36 

22.10 

93.40 

18.80 

125.20 

23.30 

Reading  Difficulty 

10.98 

.63 

11.00 

.67 

10.90 

.30 

Abstract  Knowledge 

2.39 

00 

• 

2.20 

.91 

3.20 

.93 

Expected  Attrition 

.10 

.03 

.09 

.03 

.13 

.02 

Diversity 

49.06 

43.30 

49.90 

41.10 

45.40 

45.20 

Traininq  outcome  variables8 
Quality  of  Student  Performance 

85.27 

7.60 

85.00 

7.70 

86.80 

6.90 

SIA  Time 

6.54 

15.40 

5.90 

15.20 

10.20 

15.80 

Academic  Counseling 

1.43 

3.40 

1.40 

3.40 

1.60 

3.40 

Nonacademic  Counseling 

.17 

1.50 

.19 

1.60 

.08 

• 

00 

<T* 

Retraining 

11.12 

51.50 

8.80 

42.70 

24.40 

84.50 

Academic  Attrition 

.03 

.16 

.02 

.15 

.05 

.22 

Nonacademic  Attrition 

.005 

.06 

.004 

.06 

.011 

.03 

I  Statistics  based  on  subject  samples  (Indicated  at  top  of  page). 

t  Statistics  based  on  course  samples  (Total  Sample  =  48  courses.  Model -Development  Sample  = 

|  39  courses,  Cross-Validation  Sample  =  9  courses). 

i 

t 
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model -development  sample  but  not  In  the  cross-validation  sample.  This  Is  evidenced  by  the 
sizable  standard  deviation  associated  with  yearly  student  flow  In  the  total  and  model-development 
samples  but  not  the  cross-validation  sample. 

In  examining  the  similarities  and  differences  among  the  means  and  standard  deviations  of  the 
course  content  variables  across  the  model -development  and  cross-validation  samples,  another 
Interesting  finding  emerged.  It  was  found  that  course  length,  expected  attrition  rates,  abstract 
knowledge  requirements,  and  occupational  difficulty  were  all  higher  In  the  cross-validation 
sample  than  in  the  model-development  sample.  This  confirmed  the  observation  made  with  respect  to 
the  student  Input  variables  that,  on  the  whole,  the  cross-validation  courses  are  more  difficult 
than  the  model-development  courses. 

Having  reviewed  the  descriptive  characteristics  of  the  student  Input  and  course  content 
variables,  the  results  obtained  for  the  training  outcome  variables  will  now  be  considered.  It  Is 
apparent  In  Table  3  that  the  Incidence  of  both  academic  and  nonacademic  attrition  In  the  various 
technical  training  courses  is  extremely  low.  In  the  total  sample,  only  167  academic  attritions 
and  24  nonacademic  attritions  occurred.  It  should  be  recognized  that  the  Infrequency  of  these 
outcomes  makes  prediction  much  more  difficult.  Academic  and  nonacademic  counseling  occurred  far 
more  frequently  than  did  attrition.  The  average  trainee  could  be  expected  to  receive  one  and 
one-half  academic  counseling  sessions,  whereas  one  student  In  six  could  expect  to  receive 
nonacademic  counseling.  Given  these  academic  and  nonacademic  attrition  and  counseling  rates.  It 
seems  reasonable  to  conclude  that  academic  concerns  are  far  more  likely  to  result  In  negative 
training  outcomes  than  are  nonacademic  concerns  such  as  motivation  or  attitude.  Finally,  It 
should  be  noted  that  the  relatively  large  standard  deviation  associated  with  these  variables  mty 
be  attributed  to  the  fact  that  only  a  few  students  received  counseling  or  were  eliminated  from 
training. 

Overall,  the  quality  of  student  performance  In  the  various  courses  was  adequate.  Given  a 
range  of  passing  grades  of  70  to  100,  most  students  obtained  a  grade  of  85.  SIA  appeared  to  be 
employed  frequently  In  remediating  errors,  such  that  the  typical  student  accrued  an  additional 
6.5  hours  of  one-on-one  Instruction.  However,  the  magnitude  of  the  standard  deviation  Indicates 
that  students  in  some  courses  received  substantially  more  SIA  than  did  others.  Retraining  hours, 
while  quite  high  at  a  mean  of  11.1  hours  In  the  total  sample,  also  appear  to  be  associated  with  a 
few  students  who  received  substantial  amounts  of  retraining.  This  conclusion  Is  supported  by  the 
sizable  standard  deviation  for  this  training  outcome.  Finally,  it  should  be  noted  that  the 
higher  average  attrition  rates,  retraining  time,  and  SIA  time  observed  In  the  cross-validation 
sample  relative  to  the  model-development  sample  suggest  that  the  courses  included  In  the 
cross-validation  sample  were  somewhat  more  demanding  than  those  found  in  the  model-development 
sample. 


Correlational  Data 

Revision  and  extension  of  the  initial  conceptual  model  was  based  on  Information  obtained 
during  the  Interviews  and  on  statistical  relationships  found  for  all  pairs  of  variables  in  the 
model-development  sample.  Consequently,  a  discussion  of  these  relationships  Is  In  order. 
Bivariate  correlations  derived  on  the  basis  of  data  obtained  from  the  model-development  sample 
are  presented  In  Table  4.  Correlations  derived  on  the  basis  of  data  obtained  from  the  total 
sample  and  the  cross-validation  sample  are  presented  In  Appendices  H  and  I. 

Turning  first  to  the  interrelationships  among  the  training  outcome  variables,  the  assessed 
quality  of  academic  performance  exhibited  a  pattern  of  negative  relationships  with  all  other 
training  outcomes,  as  would  be  expected  given  the  impact  of  performance  on  decisions  related  to 
counseling,  retraining,  and  attrition.  The  strongest  relationships  with  assessed  quality  were 


Table  4.  Intercorrelations  of  Student  Input,  Course  Content,  and 
Training  Outcome  Variables  In  the  Model-Development  Sample 
(5,091  students;  39  courses) 
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produced  by  academic  and  nonacademic  counseling,  respectively.  Academic  counseling,  in  turn, 
yielded  strong  positive  relationships  with  academic  attrition  and  retraining  time,  whereas 

academic  attrition  and  retraining  time  in  turn  displayed  a  strong  positive  relationship.  This 
result  was  expected  because  academic  counseling  necessarily  precedes  attrition  or  retraining 
decisions.  Similarly,  a  strong  positive  relationship  was  obtained  between  nonacademic  counseling 
and  nonacademic  attrition,  and  a  moderate  positive  relationship  was  observed  between  academic  and 
nonacademic  counseling.  As  might  be  expected  on  the  basis  of  the  coding  procedures  used, 
academic  and  nonacademic  attrition  were  not  correlated.  A  surprising  finding  was  that  SIA  time 
was  not  strongly  related  to  any  other  training  outcome  variable  except  academic  counseling.  The 
weak  relationships  found  for  SIA  time  and  other  training  outcomes  may  be  attributed  to  the  fact 
that  SIA  is  used  as  a  remediation  technique  and  is  related  to  other  outcomes  only  to  the  extent 
that  it  reveals  underlying  problems  which  cause  students  to  be  referred  for  academic  counseling. 

The  most  striking  relationships  observed  for  training  outcome  variables  was  that  final 
outcomes  like  counseling  (nonacademic  and  academic)  and  attrition  (nonacademic  and  academic) 

exhibited  weak  relationships  with  the  antecedent  student  input  and  course  content  variables, 
whereas  immediate  outcomes  such  as  assessed  quality  and  SIA  time  displayed  some  moderate 
relationships  with  the  antecedent  student  input  and  course  content  variables.  This  suggests  that 
any  effect  student  input  and  course  content  variables  have  on  final  outcome  variables  (i.e., 

counseling  and  attrition)  is  moderated  by  the  immediate  outcomes,  SIA  time  and  assessed  quality 

of  performance.  Assessed  quality  of  performance  exhibited  only  moderate  positive  relationships 
with  the  ASVAB  aptitude  composites.  However,  such  relationships  were  expected  because 
correlational  analyses  were  conducted  by  combining  courses  of  widely  varying  difficulty.  It  was 
also  found  that  age,  academic  motivation,  and  educational  level  exhibited  a  pattern  of  positive 
relationships  with  assessed  quality  of  performance.  As  expected,  motivation,  maturity,  and 
educational  level  enhance  training  performance.  The  weak  negative  relationships  between  assessed 
quality  and  the  two  interest  measures  are  counter-intuitive  and  may  reflect  inadequacies  in  these 
measures.  The  absence  of  a  relationship  between  educational  preparation  and  assessed  quality  may 
be  attributed  to  the  moderating  influence  of  other  student  input  or  course  content  variables. 

Among  the  course  content  variables,  both  instructor  quality  and  Instructor  expenenct  were 
positively  related  to  assessed  quality  of  performance.  This  finding  supports  the  notion  that 
effective  teaching  enhances  student  performance.  It  was  found  that  academic  day  length  and 
student-faculty  ratio  were  negatively  related  to  assessed  quality  of  performance.  These  results 
appear  to  support  suggestions  by  interviewees  that  an  8-hour  academic  day  is  too  much  for 
students  to  handle,  and  that  student  performance  decreases  when  Instructors  are  burdened  with  too 
large  a  class  to  devote  sufficient  time  to  individual  students.  Amount  of  feedback  was 
positively  related  to  assessed  quality  of  performance,  supporting  the  intuitive  notion  that 
frequent  feedback  increases  training  performance.  Amount  of  practice  exhibited  a  negative 
relationship  with  assessed  quality  of  performance.  The  reason  for  this  unexpected  result  is  not 
clear,  but  it  could  be  due  to  the  moderating  effect  of  other  variables.  Student  flow  exhibited  a 
negative  relationship  with  assessed  quality  of  performance.  This  result  was  expected  because 
high-flow  courses  are  often  associated  with  specialties  having  low  aptitude  requirements  and 
because  high-flow  courses  often  have  poor  student-faculty  ratios.  It  was  found  that  course 
length  exhibited  a  moderate  positive  relationship  with  assessed  quality  of  performance.  This 
finding  supports  the  notion  that  training  performance  Improves  when  students  are  given  more  time 
to  learn.  It  was  surprising  to  find  that  course  diversity,  abstract  knowledge,  and  reading 
difficulty  exhibited  a  pattern  of  positive  relationships  with  assessed  quality  of  performance. 
However,  this  result  might  be  attributed  to  the  tendency  for  more  demanding  courses  to  receive 
higher  aptitude  students.  Such  a  hypothesis  is  supported  by  the  correlations  between  these  same 
variables  and  the  ASVAB  aptitude  composites. 


Few  strong  correlations  Mere  obtained  between  S1A  time  and  student  Input  and  course  content 
variables.  It  was  found  that  SIA  tine  was  positively  related  to  course  length,  as  Might  be 
expected  since  longer  courses  would  provide  greater  opportunity  for  SIA.  It  was  also  found  that 
SIA  time  was  negatively  related  to  student  flow,  as  was  expected  given  the  fact  that  the 
instructional  demands  created  by  a  large  class  would  tend  to  decrease  the  likelihood  of  SIA. 
Abstract  knowledge,  occupational  difficulty,  diversity,  and  expected  attrition  all  yielded 
positive  relationships  with  SIA  time.  These  relationships  may  Indicate  that  SIA  time  is  likely 
to  be  higher  in  more  difficult  training  courses.  Use  of  Instructional  aids  and  hands-on  practice 
were  also  found  to  exhibit  positive  relationships  with  SIA  time,  perhaps  because  the  use  of  aids 
and  hands-on  performance  may  provide  an  overt  signal  to  Instructors  that  could  alert  them  to  the 
need  for  SIA.  Finally,  it  should  be  noted  that  SIA  time  was  effectively  unrelated  to  student 
input  variables,  suggesting  that  the  effects  of  student  attributes  on  SIA  time  may  be  moderated 
by  the  assessed  quality  of  performance  or  course  content  variables. 

In  examining  the  interrelationships  among  student  input  variables,  a  number  of  interesting 
findings  emerged.  First,  score,  on  the  ASVAB  aptitude  composites  displayed  a  pattern  of  positive 
interrelationships,  which  is  due  to  the  fact  that  some  aptitude  composites  share  common 
subtests.  Similarly,  the  aptitude  measures  displayed  positive  relationships  with  reading  level 
as  measured  by  the  AFRAT.  Scores  on  all  of  the  aptitude  composites  displayed  a  pattern  of 
positive  relationships  with  educational  level,  educational  preparation,  academic  motivation,  and 
age.  Although  these  relationships  were  expected,  given  the  tendency  of  aptitude  to  facilitate 
academic  motivation,  encourage  education,  and  maximize  time  spent  in  school,  it  should  be  noted 
that  the  strength  of  these  relationships  did  not  indicate  a  one-to-one  correspondence.  Finally, 
it  should  be  noted  that  although  a  strong  positive  relationship  was  observed  between  the  two 
interest  measures,  both  measures  exhibited  a  pattern  of  weak  negative  relationships  with  measures 
of  aptitude,  academic  motivation,  educational  level,  educational  preparation,  and  age.  This 
unexpected  finding  may  be  interpreted  as  a  side  effect  of  the  personnel  allocation  system  wherein 
the  more  talented  individuals  are  most  in  demand,  and  therefore  may  be  more  likely  to  be  given 
assignments  they  did  not  expressly  request. 

In  examining  the  correlations  between  student  input  and  course  content  variables,  only 
trivial  relationships  were  found  for  the  most  part.  However,  these  two  classes  of  variables  are 
not  independent  within  the  context  of  initial-skills  training.  Scores  on  the  various  aptitude 
measures  and  students'  reading  levels  exhibited  a  pattern  of  positive  relationships  with  course 
length,  course  reading  difficulty,  abstract  knowledge  requirements,  diversity,  and  expected 
attrition  rates.  This  pattern  of  results  would  seem  to  Indicate  that  high-ability  individuals 
are  more  likely  to  be  assigned  to  specialties  having  demanding  training  programs. 

Upon  examination  of  the  interrelationships  among  the  various  course  content  variables,  a 
complex  pattern  of  relationships  emerged.  It  was  found  that  occupational  difficulty,  course 
reading  difficulty,  diversity,  abstract  knowledge  requirements,  course  length,  and  expected 
attrition  displayed  the  strongest  Interrelationships  of  all  course  content  variables.  Due  to  the 
similarity  of  the  variables,  it  seems  reasonable  to  hypothesize  that  they  represent  a  factor 
which  reflects  course  subject-matter  difficulty. 

One  course  content  variable  that  exhibited  a  conspicuous  pattern  of  strong-to-moderate 
relationships  was  yearly  student  flow.  Student  flow  was  found  to  yield  negative  relationships 
with  occupational  difficulty  as  well  as  course  reading  difficulty,  diversity,  abstract  knowledge, 
course  length,  and  expected  attrition.  These  results  appear  to  Indicate  that  relatively  fewer 
students  enter  the  more  difficult  training  courses.  The  negative  relationships  between  student 
flow  and  both  instructor  quality  and  instructor  experience  may  be  attributed  to  the  difficulty  of 
obtaining  sufficient  numbers  of  experienced,  quality  Instructors  when  the  class  size  is  routinely 
large.  The  positive  relationship  between  student  flow  and  student- faculty  ratio  probably 


reflects  the  Influence  of  student  flow  on  class  size.  The  positive  relationship  between  flow  and 
acadeaic  day  length  suggests  that  the  less  difficult,  high-flow  courses  would  have  longer 
academic  training  days.  The  positive  relationships  obtained  between  flow  and  both  instructional 
aids  and  hands-on  Instruction  may  be  due  to  the  tendency  for  investment  in  training  support  to  be 
economically  more  feasible  when  a  large  number  of  individuals  are  to  be  trained.  Student  flow 
was  negatively  related  to  feedback,  suggesting  that  student  evaluation  may  be  less  frequent  when 
a  large  number  of  students  must  be  trained.  On  the  other  hand,  student  flow  was  positively 
related  to  practice,  indicating  that  high-flow  courses  typically  devote  more  training  hours  to 
each  unit  of  instruction.  Flow  also  exhibited  a  strong  positive  relationship  with  reenlistment 
bonus,  as  was  expected  given  that  manning  needs  underlie  both  selective  reenlistment  bonuses  and 
student  flow.  In  summary,  the  large  number  of  strong  relationships  between  student  flow  and 
other  course  content  variables  indicate  that  student  flow  plays  a  central  role  in  the  design  of 
training  courses. 

Although  practice  yielded  a  strong  positive  relationship  with  flow,  it  exhibited  a 
substantial  negative  relationship  with  feedback.  The  negative  relationship  with  feedback  was 
expected  in  view  of  the  fact  that  feedback  in  the  form  of  progress  checks,  performance  tests,  or 
sampling  written  tests  are  required  by  instructional  unit.  The  shorter  these  units  over  the 
duration  of  the  course,  the  greater  the  feedback.  Practice  was  positively  related  to 
instructional  day  length  since  a  longer  instructional  ddy  provides  for  more  practice  per  unit  of 
instruction.  It  was  found  that  Instructor  quality  and  instructor  experience  were  positively 
related  to  practice  and  negatively  related  to  feedback,  perhaps  because  with  long  instructional 
units  and  limited  feedback,  there  is  less  need  for  experienced,  quality  instructors.  Confirming 
this  hypothesis,  feedback  was  found  to  be  strongly  related  to  instructor  quality.  It  was  also 
found  that  neither  feedback  nor  practice  produced  a  systematic  pattern  of  relationships  with 
course  subject-matter  difficulty  indices,  such  as  the  reading  difficulty  of  course  materials. 

Frequency  of  use  of  instructional  aids  was  negatively  related  to  student-faculty  ratio.  This 
finding  may  be  attributed  to  the  fact  that  high  student-faculty  ratios  may  limit  the  amount  of 
instructor  supervision  that  can  be  devoted  to  the  use  of  aids.  The  use  of  aids  was  highly 
correlated  with  hands-on  practice.  This  relationship  was  not  surprising  in  view  of  the  mafM.cr  in 
which  the  instructional  aids  variable  was  defined  and  measured. 

No  systematic  relationship  was  found  between  instructor  quality  and  instructor  experience; 
however,  instructor  experience  was  related  to  the  occupational  difficulty  and  indices  of  course 
subject-matter  difficulty.  Apparently,  more  experienced  instructors  are  assigned  to  the  more 
difficult  courses.  Both  instructor  quality  and  instructor  experience  were  negatively  related  to 
student- faculty  ratio.  This  result  may  indicate  that  difficult  courses  to  which  more  experienced 
instructors  are  assigned  are  more  likely  to  have  fewer  students  per  class.  The  appropriateness 
of  this  interpretation  is  supported  by  the  negative  correlation  between  instructor  experience  and 
instructional  day  length  which  also  appears  to  be  influenced  by  student  flow.  It  should  also  be 
noted  that  student-faculty  ratio  and  day  length  were  positively  related,  perhaps  due  to  the 
influence  of  student  flow.  Finally,  both  student-faculty  ratio  and  day  length  were  inversely 
related  to  occupational  difficulty  and  indices  of  course  subject-matter  difficulty,  perhaps 
because  longer  instructional  days  and  larger  classes  are  less  likely  to  be  appropriate  as  the 
difficulty  of  the  training  material  increases. 


model  Refinement 

Earlier  in  this  report,  a  rather  simple  conceptual  model  of  the  relationship  among  student 
inputs,  course  content  variables,  and  training  outcomes  was  presented.  However,  until  the 
variables  within  each  of  these  categories  had  been  specified  and  interrelated,  it  was  difficult 
to  specify  the  model  in  greater  detail.  Having  examined  these  relationships,  it  became  possible 


to  construct  a  more  detailed  conceptual  model  of  the  relationships  among  the  student  Inputs, 
course  content  variables,  and  training  outcomes.  Because  this  elaboration  of  the  conceptual 
model  is  a  critical  step  in  the  approach  being  followed  in  the  present  effort,  this  refinement  of 
the  conceptual  model  in  relation  to  the  intercorrelations  among  the  student  input,  course 
content,  and  training  outcome  variables  will  be  presented  in  some  detail. 

As  noted  earlier,  academic  counseling,  nonacademic  counseling,  retraining,  academic 
attrition,  and  nonacademic  attrition  consistently  exhibited  weak  relationships  with  student  input 
and  course  content  variables,  whereas  assessed  quality  of  performance  and  SIA  time  exhibited  some 
moderate  relationship  with  student  input  and  course  content  variables.  This  observation 
suggested  that  the  effect  of  student  input  and  course  content  variables  on  training  outcomes  must 
operate  through  the  assessed  quality  of  student  performance  and  SIA  time.  Moreover,  the  observed 
correlations  support  the  common-sense  notion  that  assessed  quality  of  performance  has  a  direct 
causal  impact  on  both  academic  counseling  and  nonacademic  counseling,  while  SIA  time  has  a  direct 
causal  impact  only  on  academic  counseling.  Further,  it  seems  reasonable  to  assume  that  assessed 
quality  of  performance  would  have  a  direct  causal  Impact  on  SIA  time,  although  it  is  not  expected 
that  this  effect  is  strong  in  view  of  the  weak  relationship  between  these  variables. 

Given  this  conceptualization  of  the  iimediate  training  outcomes,  academic  counseling  and 
nonacademic  counseling  must  be  viewed  as  moderating  the  relationship  between  assessed  quality  and 
distal  training  outcomes  such  as  retraining  and  attrition.  This  seems  well  justified  because 
interview  data  indicated  that  either  academic  or  nonacademic  counseling  necessarily  precedes 
student  elimination  and  retraining  decisions.  Thus,  academic  counseling  was  viewed  as  having  a 
direct  causal  Impact  on  academic  attrition  and  retraining  while  nonacademic  counseling  was  held 
to  have  a  direct  causal  impact  on  nonacademic  attrition.  Because  the  interviewees  also  reported 
that  frequent  retraining  usually  led  to  academic  attrition  and  because  retraining  and  academic 
attrition  were  highly  correlated,  it  was  held  that  retraining  would  have  a  direct  causal  Impact 
on  academic  attrition.  It  was  also  assumed  that  academic  outcomes  such  as  retraining  and 
academic  attrition  would  have  no  direct  causal  effect  on  nonacademic  attrition.  This  assumption 
seems  appropriate  in  view  of  the  observed  relationships  and  interview  data  which  indicated  that 
retraining  was  an  academic  action  unrelated  to  nonacademic  Issues.  However,  It  was  held  that 
academic  counseling  might  have  a  direct  causal  impact  on  nonacademic  counseling.  This  hypothesis 
was  in  part  based  on  the  observed  pattern  of  relationships,  and  in  part  on  the  assumption  that 
academic  counseling  might  occasionally  uncover  nonacademic  problems. 

The  preceding  discussion  specified  hypotheses  concerning  the  structure  of  the  relationships 
among  training  outcomes.  At  this  point,  it  is  appropriate  to  turn  to  the  expected  relationships 
between  the  student  input  and  training  outcome  variables.  At  the  risk  of  introducing 
specification  error,  it  was  decided  that  the  two  vocational  interest  measures  should  not  be 
included  in  the  refined  model.  This  decision  was  based  on  the  ambiguity  pertaining  to  the 
validity  of  the  Interest  measures  and  the  unexplained  negative  correlations  between  the  Interest 
measures  and  assessed  quality  of  performance.  Once  this  decision  had  been  made,  the  pattern  of 
the  observed  relationships  between  student  inputs  and  training  outcomes  suggested  that  all  of  the 
remaining  student  input  variables,  (e.g.,  aptitude,  reading  level,  academic  motivation, 
educational  level,  educational  preparation,  and  age)  should  be  assumed  to  have  direct  Impacts  on 
assessed  quality  of  student  performance.  Based  on  the  observed  weak  relationships  between 
student  input  variables  and  SIA,  direct  Impacts  on  SIA  time  were  not  assumed.  Rather,  It  was 
held  that  any  impact  student  inputs  had  on  SIA  time  was  moderated  through  assessed  quality  of 
student  performance  or  the  course  content  variables. 

Having  specified  the  relationships  among  the  student  Ir.jut  and  the  training  outcome 
variables,  attention  will  now  turn  to  the  relationships  specified  for  the  course  content 
variables.  In  the  preceding  discussion,  it  was  suggested  that  student  flow  and  course 
subject-matter  difficulty  were  prime  determinants  of  the  relationships  observed  among  course 


content  variables,  such  as  student-faculty  ratio  and  instructional  day  length.  When  this 
observation  is  coupled  with  the  fact  that  training  must  consider  the  learning  difficulty  of 
occupational  tasks,  it  appears  reasonable  to  assert  that  training  is  driven  by  the  difficulty  of 
occupational  tasks,  the  difficulty  of  course  subject  matter,  and  manpower  requirements. 

Although  occupational  difficulty  could  be  measured  directly,  manpower  requirements  and  course 
subject-matter  difficulty  represented  latent  variables  that  were  defined  through  observed 
variables.  The  high  correlation  found  between  student  flow  and  reenlistment  bonus  suggested  that 
manpower  requirements  should  be  defined  by  these  two  variables.  Given  the  pattern  of 
correlations  discussed  earlier,  it  seemed  that  course  subject-matter  difficulty  could  be  defined 
by  the  reading  difficulty  of  course  materials,  the  diversity  of  course  materials,  abstract 
knowledge  requirements,  expected  attrition  rates,  and  course  length.  However,  it  was  held  that 
only  the  former  four  variables  would  be  used  to  define  subject-matter  difficulty  since  course 
length  is  reasonably  considered  an  outcome  of  course  subject-matter  difficulty. 

According  to  this  conceptualization,  course  subject-matter  difficulty,  occupational 
difficulty,  and  manpower  requirements  were  held  to  be  the  three  fundamental  or  primal  variables 
affecting  training  design.  These  variables  were  assumed  to  be  correlated  with  each  other.  Also, 
these  three  primal  variables  were  held  to  determine  lower-order  course  variables,  such  as 
instructional  day  length  and  student-faculty  ratio.  Thus,  course  length,  day  length, 
student-faculty  ratio,  instructor  quality.  Instructor  experience,  use  of  aids,  hands-on  practice, 
feedback,  and  practice  were  held  to  be  determined  by  these  three  variables.  However,  based  on 
the  observed  correlations,  it  was  decided  that  instructional  aids  and  hands-on  practice  should  be 
combined  into  a  latent  variable  representing  instructional  support,  whereas  feedback  and  practice 
should  be  combined  into  a  latent  variable  representing  feedback  intensity. 

Feedback  intensity,  instructional  support,  instructor  experience,  instructor  quality, 
student-faculty  ratio,  day  length,  and  course  length  were  all  assumed  to  be  capable  of  having  a 
direct  causal  impact  on  both  the  assessed  quality  of  student  performance  and  SIA  time.  The 
direct  causal  relationships  with  SIA  time  were  postulated  for  these  course  variables,  unlike  the 
student  inputs,  because  it  was  felt  that  course  content  variables  such  as  course  length  or  day 
length  could  affect  the  feasibility  of  SIA  time  regardless  of  performance.  Further,  it  was 
assumed  that  independent  of  the  effects  of  the  various  course  variables  on  quality  of 
performance,  subject-matter  difficulty,  occupational  difficulty,  and  manpower  requirements  might 
also  have  a  direct  causal  impact  on  both  assessed  quality  of  performance  and  SIA  time. 

A  schematic  representation  of  the  refined  model  is  presented  in  Figure  3.  As  may  be  seen, 
the  student  input  variables  are  assumed  to  be  intercorrelated  and  have  a  direct  causal  impact  on 
the  assessed  quality  of  performance.  Course  subject-matter  difficulty,  occupational  difficulty, 
and  manpower  requirements  are  assumed  to  have  a  direct  causal  impact  on  all  other  course 
variables  as  well  as  on  SIA  time  and  the  assessed  quality  of  performance.  Finally,  all 
lower-order  course  variables,  such  as  day  length  or  student-faculty  ratio,  are  assumed  to  have  a 
direct  causal  Impact  on  both  the  assessed  quality  of  performance  and  SIA  time. 

The  causal  paths  above  the  course  parameters  reflect  the  hypothesized  effects  of  the  three 
primal  variables  in  order  from  left  to  right,  while  the  causal  paths  below  the  course  parameters 
reflect  their  effects  on  quality  of  performance  and  SIA  time,  respectively.  Finally,  the  diagram 
shows  that  assessed  quality  of  performance  and  SIA  time  determine  all  other  training  outcomes 
when  moderated  through  academic  and  nonacademic  counseling. 
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Modeling  Results 

When  the  refined  model  was  analyzed  by  LISREL  V,  it  was  found  to  provide  an  excellent  fit  to 
the  observed  relationships  among  the  student  input,  course  content,  and  training  outcome 
variables  within  the  model-development  sample.  Overall,  it  was  found  that  the  model  yielded  a 
goodness-of-f it  index  of  .59,  and  a  residual  of  .19.  This  goodness-of-f it  index  indicates  that 
the  refined  model  provided  an  unusually  close  fit  to  the  observed  data.  This  was  confirmed  by 
the  residual  term,  which  indicated  that  only  19X  of  the  variance  in  the  relationships  among  the 
three  classes  of  variables  could  not  be  accounted  for  by  the  hypothetical  model. 

Evidence  of  the  predictive  power  of  the  model  may  be  obtained  by  considering  the  Rs  generated 
by  the  model  against  each  of  the  dependent  variables.  These  data  are  presented  in  Table  5  and 
broken  down  with  respect  to  whether  the  dependent  variable  was  a  training  outcome  measure  or  a 
predicted  course  variable.  For  the  training  outcome  variables,  the  model  yielded  an  R  of  .75 
against  assessed  quality  of  performance.  Multiple  correlation  coefficients  of  .60  and  .50  were 
obtained  against  academic  and  nonacademic  counseling,  respectively;  retraining  time,  academic 
attrition,  and  nonacademic  attrition  produced  Rs  of  .76,  .83,  and  .59,  respectively.  Overall, 
this  pattern  of  results  indicates  excellent  prediction  of  training  outcomes,  with  academic 
outcome  variables  being  somewhat  better  predicted  than  nonacademic  outcomes  such  as  nonacademic 
counseling  and  nonacademic  attrition.  The  weakest  prediction  of  a  training  outcome  variable  was 
obtained  for  SIA  time,  which  yielded  an  R  of  .35.  Apparently,  SIA  time  represents  a  somewhat 
unique  variable  among  the  training  outcomes. 

Table  5.  Multiple  Rs  and  R^s  for  Training  Outcome  and 
Course  Content  Variables 


R 

Training  Outcome  Variables 

Assessed  Quality  of  Performance 

.75 

.54 

SIA  Time 

.35 

.12 

Academic  Counseling 

.60 

.36 

Nonacademic  Counseling 

.50 

.25 

Retraining  Time 

.76 

.59 

Academic  Attrition 

.83 

.70 

Nonacademic  Attrition 

.59 

.35 

Predicted  Course  Variables 

Course  Length 

.86 

.75 

Day  Length 

.60 

.36 

Student-Faculty  Ratio 

.62 

.38 

Instructor  Quality 

.58 

.34 

Instructor  Experience 

.60 

.36 

Instructional  Support 

.54 

.30 

Feedback  Intensity 

.68 

.47 

For  the  course  variables  that  were  held  to  be  determined  by  course  subject-matter  difficulty, 
occupational  difficulty,  and  manpower  requirements,  a  similar  pattern  of  results  was  obtained. 
These  three  variables  yielded  Rs  against  course  length,  instructional  day  length,  student-faculty 
ratio,  instructor  quality,  instructor  experience,  instructional  support,  and  feedback  intensity 
of  .86,  .60,  .62,  .58,  .60,  .54,  and  .68,  respectively.  Overall,  the  magnitude  of  these  Rs 
Indicates  that  an  optimal  combination  of  subject-matter  difficulty,  occupational  difficulty,  and 
manpower  requirements  did  an  excellent  job  of  predicting  the  lower-order  course  variables.  This 


is  an  important  finding  because  it  suggests  that  it  is  fully  appropriate  to  conceive  of  training 
as  being  driven  by  course  subject-matter  difficulty,  the  difficulty  of  occupational  tasks,  and 
manpower  requirements. 

The  foregoing  results  indicated  that  the  refined  model  was  capable  of  accounting  for  the 
relationships  observed  among  the  student  input,  course  content,  and  training  outcome  variables, 
as  well  as  yielding  excellent  prediction  of  training  outcomes.  Given  this  observation,  it  would 
seem  appropriate  to  examine  the  nature  and  magnitude  of  the  standardized  path  coefficients 
generated  by  the  LISREL  V  program  in  optimizing  regeneration  of  the  variance-covariance  matrix. 
The  standardized  path  coefficients  generated  by  the  LISREL  V  program  are  presented  in  Figure  4. 
A  standardized  path  coefficient  represents  the  unique  effects  of  an  independent  variable  on  a 
dependent  variable  when  both  are  expressed  in  standard  score  form.  It  can  be  interpreted  as  the 
magnitude  of  change  in  the  specified  dependent  variable  given  a  one-unit  change  In  the 
independent  variable.  For  example,  in  Figure  4,  the  standardized  path  coefficient  of  .16, 
occurring  above  the  arrow  connecting  aptitude  with  quality  of  performance,  indicates  that  an 
increase  in  average  quality  of  performance  (dependent  variable)  of  16/100ths  of  a  standard  score 
unit  would  be  expected  to  follow  an  increase  in  average  aptitude  (independent  variable)  of  1 
unit,  when  all  other  independent  variables  are  held  constant. 

Inspection  of  Figure  4  indicates  that  the  student  input  variables  have  substantial  effects  on 
assessed  quality  of  student  performance.  The  most  powerful  effects  were  generated  by  aptitude 
(.16),  reading  level  (.16),  and  academic  motivation  (.14).  These  results  suggest  that 
intellectual  ability  and  motivation  to  achieve  are  prime  determinants  of  training  performance. 
However,  sizable  effects  were  also  produced  by  age  (.11)  and  educational  level  (.12),  indicating 
that  maturity  and  educational  exposure  both  have  independent  causal  effects  on  the  quality  of 
student  performance.  The  weakest  path  coefficient  produced  by  the  various  student  Input 
variables  was  a  coefficient  of  .07  associated  with  educational  preparation.  This  suggests  that 
although  specific  educational  preparation  may  contribute  to  performance,  in  a  well-designed 
training  program  it  does  not  have  a  particularly  strong  influence. 

A  number  of  interesting  findings  emerged  among  the  course  content  variables.  In  defining 
course  subject-matter  difficulty,  it  appears  that  abstract  knowledge  requirements  are  the  single 
best  indicator  of  this  construct,  although  sizable  coefficients  were  also  obtained  for  reading 
difficulty,  course  diversity,  and  expected  attrition  rates,  indicating  that  all  these  variables 
must  be  considered  for  optimal  definition  of  the  construct.  Similarly,  in  defining  manpower 
requirements,  both  selective  reenlistment  bonus  and  yearly  student  flow  yielded  sizable 
coefficients.  However,  as  indicated  by  the  substantially  greater  magnitude  of  its  standarized 
path  coefficient,  yearly  student  flow  appears  to  be  a  more  appropriate  index  of  manpower 
requirements.  Manpower  requirements  and  course  subject-matter  difficulty  proved  to  be  negatively 
related,  as  was  expected  from  the  correlational  data.  Moreover,  course  subject-matter  difficulty 
was  positively  related  to  occupational  difficulty,  apparently  because  the  difficulty  of  a 
training  program  reflects  the  difficulty  of  job  tasks.  Occupational  difficulty,  however,  was 
positively  related  to  manpower  requirements,  unlike  course  subject-matter  difficulty.  These 
results  might  be  attributed  to  a  greater  civilian  demand  for  individuals  trained  in  more 
difficult  occupations,  and  the  limitations  that  difficulty  of  a  training  program  places  on 
student  flow. 

In  examining  how  course  subject-matter  difficulty,  occupational  difficulty,  and  manpower 
requirements  determined  lower-order  course  variables,  a  number  of  interesting  findings  emerged. 
It  was  found  that  course  length  was  determined  primarily  by  occupational  difficulty  (.40)  and 
subject-matter  difficulty  (.58),  rather  than  by  manpower  requirements  (.02).  This  finding  could 
be  expected  in  a  system  where  individuals  must  be  trained  to  a  constant  standard  of  performance 
regardless  of  manpower  requirements.  On  the  other  hand,  it  was  found  that  instructional  day 
length  tended  to  increase  with  increasing  manpower  requirements  (.36)  and  decrease  with 


increasing  course  subject-matter  difficulty  (-.39).  This  set  of  path  coefficients  appears  to 
support  the  hypothesis  that  an  8-hour  instructional  day  is  used  as  one  means  of  acconmodating 
greater  manpower  needs,  but  that  its  use  in  difficult  courses  will  be  offset  by  the  tendency  of  a 
longer  instructional  day  to  overload  students  with  information. 

Student-faculty  ratio  was  found  to  be  negatively  related  to  both  subject-matter  difficulty 
(-.38)  and  occupational  difficulty  (-.30),  while  it  was  virtually  unrelated  to  manpower 
requirements  (-.02).  Thus,  it  appears  that  the  number  of  students  per  instructor  is  primarily 
due  to  the  difficulty  of  the  job  and  course  subject  matter  rather  than  manning  needs.  Instructor 
quality  was  found  to  be  negatively  related  to  subject-matter  difficulty  (-.10),  occupational 
difficulty  (-.30),  and  manpower  requirements  (-.47).  Apparently,  a  high  quality  of  instruction 
is  less  feasible  when  the  class  size  is  large  and  the  course  material  is  difficult  to  teach.  On 
the  other  hand,  instructor  experience  was  found  to  be  positively  related  to  subject-matter 
difficulty  (.42)  and  occupational  difficulty  (.19).  These  results  confirmed  the  previous 
hypothesis  that  more  difficult  courses  require  and  are  typically  assigned  to  more  experienced 
instructors.  However,  it  was  also  found  that  instructor  experience  tended  to  be  limited  by 
manpower  requirements  (-.17),  indicating  that  when  need  is  high,  it  is  more  difficult  to  obtain 
highly  experienced  instructors. 

Feedback  intensity  was  conceptually  defined  by  the  amount  of  feedback  occurring  in  a  course 
and  the  amount  of  practice  provided.  This  construct  was  confirmed  by  the  relationship  of 
feedback  intensity  with  amount  of  feedback  (.85)  and  practice  (-.67).  Feedback  intensity  was  not 
influenced  by  course  subject-matter  difficulty  (.04);  however,  it  tended  to  be  limited  by  both 
occupational  difficulty  (-.20)  and  manpower  requirements  (-.48).  These  negative  effects  might  be 
attributed  to  the  fact  that  it  is  more  difficult  for  instructors  to  provide  frequent  feedback 
concerning  performance  on  difficult  tasks  and  to  the  fact  that  a  large  number  of  students  tends 
to  restrict  the  amount  of  feedback  that  can  be  provided. 

As  defined  in  the  refined  model,  instructional  support  was  considered  to  be  a  function  of  the 
number  of  instructional  aids  used  and  the  amount  of  hands-on  practice  provided.  This  hypothesis 
was  confirmed  by  the  coefficient  of  .28  obtained  for  hands-on  practice  and  the  coefficient  of  .31 
obtained  for  the  number  of  aids  in  use.  Although  the  magnitude  of  these  coefficients  suggests 
that  this  latent  variable  was  not  as  well  defined  as  the  preceding  latent  variables,  a  consistent 
and  interpretable  pattern  of  path  coefficients  for  this  construct  was  generated  by  subject-matter 
difficulty  (.44),  occupational  difficulty  (.23),  and  manpower  requirements  (.10).  As  might  be 
expected,  instructional  support  is  most  likely  to  occur  when  course  training  materials  and  job 
tasks  are  difficult.  Furthermore,  instructional  support  tends  to  be  employed  more  frequently 
when  student  flow  is  sufficiently  high  to  result  in  a  payoff  from  its  development. 

In  addition  to  the  above  interrelationships,  it  was  found  that  the  course  content  variables 
were  capable  of  determining  training  outcomes.  All  three  of  the  primal  course  content  variables 
defined  by  subject-matter  difficulty  (-.08),  occupational  difficulty  (-.13),  and  manpower 
requirements  (-.10)  had  a  direct  causal  impact  on  the  assessed  quality  of  student  performance. 
These  effects  indicate  that  difficult  courses,  difficult  job  tasks,  and  requirements  to  train 
large  numbers  of  students  will  tend  to  restrict  the  quality  of  student  performance.  It  was  also 
found  that  SIA  time  was  slightly  influenced  by  subject-matter  difficulty  (.16),  by  occupational 
difficulty  (.06),  and  by  manpower  requirements  (-.05).  In  fact,  subject-matter  difficulty  was 
the  prime  determinant  of  SIA  time,  as  was  expected,  since  more  difficult  courses  should  cause 
progress  check  failure  and  therefore  greater  SIA  time.  Alternatively,  manpower  requirements  had 
a  negative  effect  on  SIA  time  because  large  classes  tend  to  restrict  the  amount  of  SIA  that  can 
be  provided. 

When  the  effects  of  these  three  primal  variables  were  controlled,  it  was  found  that  certain 
lower-order  course  variables  could  also  have  an  impact  on  the  assessed  quality  of  student 
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performance  and  SIA  time.  In  keeping  with  the  interview  reports,  it  was  found  that  feedback 
intensity  had  the  greatest  impact  on  the  quality  of  performance  (.09)  and  SIA  time  (.08).  The 
former  relationship  may  reflect  the  generally  recognized  facilitative  effects  of  feedback  on 
performance,  whereas  the  latter  relationship  may  indicate  that  feedback  can  serve  as  an  indicator 
of  the  need  for  SIA.  As  expected,  instructor  quality  was  found  to  have  positive  effects  on  the 
quality  of  performance  (.05)  and  SIA  time  (.08).  These  results  suggested  that,  good  instruction 
enhances  student  performance  and  good  instructors  may  be  more  likely  to  provide  SIA  time. 
Finally,  as  expected,  course  length  had  moderate  positive  impacts  on  both  SIA  time  (.05)  and  the 
quality  of  student  performance  (.06). 

The  path  coefficients  obtained  for  instructional  day  length,  student-faculty  ratio, 
instructional  support,  and  instructor  experience  were  not  especially  powerful  when  the  effects  of 
course  subject-matter  difficulty,  occupational  difficulty,  and  manpower  requirements  were 
partialed  out.  However,  both  day  length  and  student-faculty  ratio  had  minor  negative  effects  on 
assessed  quality  of  performance  and  SIA  time,  supporting  the  information  overload  hypothesis  and 
the  notion  that  student  performance  had  a  slight  tendency  to  decrease  as  class  size  increased. 
Instructor  experience,  on  the  other  hand,  tended  to  have  minor  positive  effects  on  both  the 
quality  of  performance  and  SIA  time,  indicating  that  experienced  instructors  facilitate  quality 
performance  and  tend  to  provide  more  SIA.  Finally,  instructional  support  produced  a  minor 
positive  effect  for  SIA  time,  perhaps  because  such  support  provides  signals  concerning  the  need 
for  SIA. 

As  indicated  in  the  preceding  discussion,  the  assessed  quality  of  student  performance  had 
substantial  effects  on  all  relevant  training  outcomes,  especially  academic  counseling  (-.20)  and 
nonacademic  counseling  (-.23).  Academic  counseling,  in  turn,  had  a  positive  effect  on  academic 
attrition  (.22)  and  retraining  (.34),  while  nonacademic  counseling  had  a  positive  effect  on 
nonacademic  attrition  (.24).  These  results  confirm  the  earlier  hypothesis  that  counseling 
moderates  the  relationship  between  quality  of  performance  and  subsequent  training  outcomes  such 
as  retraining  and  attrition.  Further,  a  path  coefficient  between  academic  and  nonacademic 
counseling  (.15)  supported  the  proposition  that  academic  counseling  tends  to  reveal  nonacademic 
problems.  A  path  coefficient  between  retraining  time  and  academic  attrition  (.25)  was  al  >0  found 
in  this  analysis.  Thus,  high  retraining  time  appears  to  be  an  antecedent  of  attrition,  as  was 
indicated  by  the  interview  data. 

The  assessed  quality  of  performance  exhibited  a  weak  causal  impact  on  SIA  time  (-.07).  This 
suggests  that  poor  performance  might  have  some  effect  on  SIA  time,  but  that  the  difficulty  of  the 
course  material  still  appears  to  be  the  prime  determinant.  Finally,  as  expected,  SIA  time  had  a 
direct  impact  (.18)  on  academic  counseling.  This  result  confirms  the  expectation  that  repeated 
SIA  sessions  tend  to  serve  as  a  signal  for  academic  problems. 

Overall,  the  foregoing  discussion  of  the  standardized  path  coefficients  resulting  from  the 
analysis  indicates  that  the  refined  model  yields  a  highly  meaningful  and  interpretable  set  of 
interrelationships.  This  provides  compelling  evidence  for  the  validity  of  the  model  when  coupled 
with  the  observed  residual,  goodness-of-f it  test,  and  Rs.  The  refinea  model,  with  regression 
coefficients  and  total  effects,  is  presented  without  further  discussion  in  Appendices  J  and  K, 
respectively. 

One  last  topic  that  must  be  addressed  concerns  one  of  the  basic  assumptions  of  path 
modeling.  The  independence  of  errors  associated  with  variable  measurement  is  one  of  the  more 
important  assumptions  associated  with  this  analytical  technique.  Evaluation  of  correlations 
between  the  error  terms  for  the  course  variables,  as  well  as  correlations  between  the  error  terms 
of  the  course  variables  and  training  outcome  variables,  served  as  a  basis  of  determining  the 
tenability  of  this  assumption.  These  correlation  coefficients  are  provided  in  Appendices  L  and 
M,  respectively.  Inspection  of  these  data  indicates  that  the  error  terms  of  the  course 


parameters  and  the  outcome  variables  were  not  strongly  related,  and  that  the  error  terms  among 
the  course  parameters  themselves  were  effectively  Independent.  Correlations  between  the  error 
terms  for  the  training  outcome  variables  which  are  presented  in  Appendix  N  indicated  a  pattern 
which  conforms  to  the  formal  path  model.  For  instance,  errors  in  the  measurement  of  the  assessed 
quality  of  performance  were  slightly  related  to  errors  associated  with  other  training  outcomes. 
However,  this  result  was  expected  because  assessed  quality  of  performance  is  the  basis  for 
decisions  associated  with  other  training  outcomes  such  as  retraining  and  attrition.  Similarly, 
measurement  errors  for  academic  counseling,  which  plays  a  pivotal  role  in  these  training  outcome 
decisions,  were  correlated  with  all  other  variables.  Thus,  the  pattern  of  relationships  among 
error  terms  is  consistent  with  the  overall  model. 


Cross-Validation 

The  preceding  discussion  has  demonstrated  that  the  refined  model  provides  a  meaningful  and 
interpretable  description  of  the  relationships  among  student  input,  course  content,  and  training 
outcome  variables  and  that  it  accounts  for  much  of  the  variance  in  training  outcomes.  Although 
these  results  clearly  suggest  the  predictive  utility  of  the  model,  the  questions  that  arise  at 
this  point  are  whether  accurate  predictions  will  result  when  courses  not  employed  in  model 
development  are  examined,  and  most  importantly,  whether  the  refined  model  is  sufficiently 
representative  to  be  effectively  employed  in  making  predictions  for  a  diverse  set  of  courses. 

To  answer  these  questions,  a  cross-validation  analysis  compatible  with  the  path-analytic 
procedures  was  carried  out.  This  analysis  entailed  obtaining  standard  scores  on  the  student 
input  and  course  content  variables  for  each  of  nine  courses  held  for  cross-validation  and 
entering  values  for  the  student  input  and  course  content  variables  for  each  course  into  the 
relevant  regression  equations  to  obtain  standard  score  predictions  for  the  seven  training 
outcomes.  These  standard  score  predictions  were  then  converted  to  raw  score  predictions.  Once 
this  information  was  obtained,  it  was  directly  compared  to  the  raw  scores  and  standard  scores 
actually  observed  for  the  cross-validation  courses  for  each  training  outcome. 

It  should  be  noted  that  when  generating  training  outcome  predictions  for  each  course,  the 
nature  of  the  model  dictated  the  application  of  a  certain  strategy.  Distal  training  outcomes  are 
not  predicted  directly  from  knowledge  of  student  input  or  course  content  variables.  The  refined 
model  holds  that  distal  training  outcomes,  such  as  attrition  and  retraining,  are  dependent  on  the 
intervening  counseling  process  and  assessed  quality  of  performance.  Thus,  predictions  of  the 
assessed  quality  of  performance  and  SI A  time,  which  issued  directly  from  knowledge  of  student 
input  and  course  content  variables,  were  used  in  deriving  predictions  of  academic  and  nonacademic 
counseling  rates.  In  turn,  the  predicted  academic  and  nonacademic  counseling  rates  were  used  In 
conjunction  with  the  appropriate  regression  weights  to  generate  predictions  of  academic 
attrition,  nonacademic  attrition,  and  retraining  time.  Although  this  procedure  was  in  keeping 
with  the  design  of  the  model  and  its  intended  application,  it  should  be  borne  in  mind  that  due  to 
the  possibility  of  cumulative  prediction  error,  it  constitutes  a  conservative  test  of  the 
predictive  accuracy  of  the  model  with  respect  to  the  distal  training  outcomes  of  academic 
counseling,  nonacademic  counseling,  retraining  time,  academic  attrition,  and  nonacademic 
attrition. 

Table  6  displays  observed  and  predicted  scores  in  both  raw  score  and  standard  score  form  for 
all  training  outcomes  for  each  of  the  nine  cross-validation  courses.  As  may  be  seen,  predictions 
of  assessed  quality  of  performance,  which  was  expressed  as  an  average  final  course  grade, 
approximated  the  observed  values.  Across  the  nine  courses,  there  was  an  average  absolute 
difference  between  predicted  and  observed  raw  scores  of  2.34  points.  It  Is  notewortlyr  that  the 
model  predicted  average  final  school  grade  about  equally  well  for  all  nine  courses.  In  fact,  the 


m 


m 


\<A 


•}.$ 


i/i  q  1  lh  ^•c\jc\jir>roui^*  — 

0>  L*J  I  ••••••••• 

u  ocl—  — 

O  a.) 

o 

v>  | 

oolro  rsfnkncooo<-^u) 
*  col  •  .  •  . . 

•O  OlCNI  *“04  CM  CM  *“ 


-p  Iq  Ncgocomr-coroN 
L  v>|  uj  conocM'-o^fON 


»  •fUJfOCOOCM'-O^fOCsJ 

«  V  a . . 

-g  t  a 

co  oo 

*o  u  i/>  co  o  oo  *—  co  n  cvj  n  >— 

4J  l/»  CO  •—  •  CVJ  CM  M  ^  CO  «-  O 

l/>  O  •  I  ••••••• 


' 

•o 

4) 

■ 

O 

u 

c 

QJ 

4> 

Q. 

&. 

l/> 

4) 

4- 

o> 

4- 

c 

1 

Q 

4J 

4-» 

L. 

3 

a» 

r— 

o 

O 

k. 

i/l 

Q_ 

U 

-O 

< 

4) 

a> 

■M 

Ol 

<0 

t 

i. 

Q) 

O 

> 

o 

< 

4’ ,  r,y 


largest  difference  observed  between  predicted  and  observed  raw  scores  on  this  measure  was  only 
4.6  points. 

The  model's  accuracy  in  predicting  assessed  quality  of  performance  was  not  matched  with 
regard  to  SIA  time.  The  overall  difference  between  predicted  and  observed  raw  scores  for  the 
nine  cross-validation  courses  was  6.83  hours.  These  results  were  expected,  given  that  SIA  time 
produced  the  smallest  R  observed  in  the  present  investigation. 

Table  6  also  presents  the  results  obtained  in  the  cross-validation  analysis  for  academic 
counseling  and  nonacademic  counseling.  For  the  nine  courses,  the  average  difference  between 
observed  and  predicted  raw  scores  was  .67  for  academic  counseling,  and  .64  for  the  nonacademic 
counseling.  The  largest  difference  in  observed  and  predicted  raw  scores  was  .11  for  nonacademic 
counseling  and  1.5  for  academic  counseling.  Given  the  relative  infrequency  of  nonacademic 
counseling,  it  was  surprising  that  the  model  provided  more  accurate  predictions  for  it  than  for 
academic  counseling.  However,  this  result  might  be  attributed  to  the  fact  that  within  the  model, 
cumulative  errors  carried  over  from  SIA  time  would  affect  predictions  of  academic  counseling  more 
than  they  would  predictions  of  nonacademic  counseling. 

With  respect  to  retraining  time,  the  average  difference  obtained  between  predicted  and 
observed  raw  scores  was  11.6.  Predictive  accuracy  held  across  courses,  with  the  exception  of 
those  that  had  unusally  high  retraining  times.  The  results  obtained  for  academic  attrition 
indicated  that  the  average  difference  between  predicted  and  observed  attrition  rates  was  .03. 

As  was  noted  earlier,  the  nonacademic  attrition  rate  was  extremely  low.  In  fact,  for  the 
nine  cross-validation  courses,  only  one  student  was  eliminated  for  nonacademic  reasons.  This 
nonacademic  student  elimination  occurred  in  the  Jet  Engine  Mechanic  course.  Surprisingly,  the 
highest  predicted  nonacademic  attrition  rate  was  for  this  course,  and  in  fact,  it  was  the  only 
predicted  value  which  was  not  extremely  close  to  zero. 

Clearly,  the  predicted  outcomes  were  reasonably  consistent  with  the  actual  values.  However, 
the  cross-validation  study  does  not  fully  demonstrate  general izability  of  results  across  courses, 
time,  and  both  courses  and  time.  Therefore,  we  think  it  best  to  view  the  present  model  as  a 
reasonably  good  first  approximation  to  the  complex  relationships  involved  in  initial-skills 
training.  As  additional  data  become  available  and  as  personnel  and  training  managers  gain 
experience  with  the  working  of  the  system,  it  would  be  appropriate  to  revisit  empirically  the 
accuracy  of  the  model's  predictions  for  the  same  courses  at  different  points  in  time,  for 
different  courses,  and  for  different  courses  at  different  points  in  time. 


IV.  DISCUSSION 


Findings 

The  principal  concern  giving  rise  to  this  work  was  the  need  for  some  technique  that  would 
allow  estimation  of  the  effects  of  aptitude  requirement  adjustments  on  technical  training 
outcomes  prior  to  actually  making  such  adjustments.  Because  the  model  described  above  has  been 
shown  to  be  quite  stable  and  to  have  substantial  predictive  power,  it  appears  to  provide  a  viable 
basis  for  drawing  general  conclusions  concerning  the  relationship  between  aptitude  and  training 
outcomes. 

The  most  straightforward  conclusion  indicated  by  the  content  of  the  refined  model  is  that 
aptitude  is  clearly  a  prime  determinant  of  training  outcomes.  This  was  attested  to  by  the  fact 
that  aptitude  yielded  one  of  the  most  powerful  path  coefficients  against  assessed  quality  of 
student  performance.  Nevertheless,  this  observation  should  not  be  taken  as  indicating  that 
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aptitude  Is  the  only  determinant  of  training  outcomes.  The  refined  model  clearly  indicates  that 
a  variety  of  other  student  input  variables,  such  as  reading  level,  academic  motivation, 
educational  level,  and  educational  preparation  are  all  associated  with  sizable  path  coefficients, 
indicating  that  they  too  have  a  significant  impact  on  assessed  quality  of  student  performance. 
This  is  not  an  especially  surprising  finding  since  a  wealth  of  literature  exists  indicating  that 
motivation  to  perform  and  prior  educational  achievements  are  often  critical  determinants  of 
training  performance.  Moreover,  the  magnitude  of  the  coefficients  associated  with  academic 
motivation,  educational  level,  and  educational  preparation  seem  to  indicate  that  in  constructing 
a  manpower  allocation  system  of  maximum  effectiveness,  these  indices  of  student  quality  should  be 
taken  into  account.  Further,  it  appears  that  there  might  be  substantial  value  in  considering 
some  index  of  maturity,  such  as  age,  given  the  sizable  path  coefficient  produced  by  this 
variable.  Although  an  attempt  was  made  to  examine  interest,  the  available  measures  of  this 
variable  made  it  impossible  to  effectively  assess  the  impact  of  interest  on  the  assessed  quality 
of  student  performance.  Thus,  this  issue  must  remain  open  until  more  effective  and  appropriate 
interest  measures  become  available. 

In  addition  to  recognizing  that  aptitude  is  only  one  of  a  number  of  variables  that  influence 
the  quality  of  student  performance,  it  must  also  be  recognized  that  aptitude  and  its  effects  are 
in  part  imbedded  in  relationships  with  a  number  of  other  variables.  Thus,  when  considering  the 
potential  impact  of  aptitude  requirement  adjustments,  one  must  view  these  effects  as  being 
complex,  multifaceted  effects  that  are  exerted  through  a  variety  of  closely  related  variables. 

One  of  the  more  Interesting  and  significant  findings  generated  by  the  model  is  the  manner  in 
which  aptitude,  or  for  that  matter  any  student  input  variable,  influences  training  outcomes. 
Aptitude  and  other  student  inputs  have  a  strong  direct  impact  on  the  assessed  quality  of  student 
performance.  However,  the  model  indicates  that  student  inputs  have  no  direct  effect  on  distal 
training  outcomes  such  as  academic  attrition,  nonacademic  attrition,  and  retraining.  Rather, 
student  inputs  affect  the  assessed  quality  of  student  performance,  which  in  turn  affects  either 
academic  counseling  or  nonacademic  counseling.  Retraining  time,  academic  attrition,  and 
nonacademic  attrition  represent  potential  negative  outcomes  of  these  counseling  sessions. 

This  set  of  findings  appears  to  conform  to  current  operational  procedures  as  they  occur  in 
the  initial-skills  training  programs  administered  by  ATC.  For  example,  the  fundamental 
importance  of  academic  and  nonacademic  counseling  in  affecting  distal  training  outcomes  is  an 
Important  finding  which  has  a  number  of  implications.  First,  it  indicates  that  whether  an 
individual  is  eliminated  from  training  or  retrained  is  ultimately  in  the  hands  of  course 
Instructors.  Second,  given  the  fundamental  importance  of  instructors’  counseling  decisions  in 
determining  retraining  and  attrition,  it  appears  that  more  attention  should  be  given  to  this 
process  in  future  research  efforts,  and  that  both  academic  and  nonacademic  counseling  should  be 
considered  major  training  outcomes.  Third,  given  the  apparent  importance  of  this  process  in 
mediating  the  relationship  between  performance  and  distal  training  outcomes,  considerations 
underlying  trainers'  counseling  decisions  should  receive  attention  not  only  in  research  efforts, 
but  also  with  regard  to  policy  formulation. 

Two  other  points  concerning  the  relationships  observed  among  training  outcome  variables 
should  be  elaborated.  First,  although  student  inputs  affect  assessed  quality  of  performance, 
which  in  turn  is  the  fundamental  determinant  of  counseling  and  attrition,  the  model  indicates 
that  a  clear-cut  distinction  is  made  between  purely  academic  performance  problems  and  academic 
performance  problems  resulting  from  nonacademic  causes  such  as  a  lack  of  motivation.  Thus,  a 
careful  distinction  should  be  made  between  academic  and  nonacademic  outcomes  despite  their  common 
cause.  It  should  be  noted  that  the  present  findings  suggest  that  academic  problems  are  far  more 
common  than  nonacademic  problems,  and  that  academic  counseling  at  times  reveals  nonacademic 
problems  as  Indicated  by  the  causal  path  from  academic  to  nonacademic  counseling. 


The  second  point  is  that  while  S2A  time  represents  a  significant  outcome  variable,  it  is 
somewhat  unique.  S1A  time  was  not  found  to  be  highly  related  to  either  student  inputs  or  the 
assessed  quality  of  student  performance.  Rather,  SIA  time  was  primarily  driven  by  the  difficulty 
of  course  subject  matter  and  occupational  difficulty.  Thus,  it  appears  that  SIA  time  is  truly 
employed  as  a  remediation  vehicle.  However,  the  model  also  indicates  that  a  repeated  need  for 
remediation  may  affect  academic  counseling  and,  in  turn,  academic  attrition  and  retraining 
decisions.  This  suggests  that  though  SIA  is  a  somewhat  unique  variable,  it  is  of  some  importance 
to  conceptualization  and  prediction  of  training  outcomes. 

As  was  Implied  in  the  discussion  of  SIA  time,  the  various  course  content  variables  also 
produced  some  interesting  relationships.  Broadly  speaking,  the  refined  model  hypothesizes  that 
training  is  driven  by  three  primal  variables:  course  subject-matter  difficulty,  occupational 
difficulty,  and  manpower  requirements.  That  is,  the  model  holds  that  courses  are  designed  on  the 
basis  of  the  difficulty  of  the  job,  the  difficulty  of  course  training  materials,  and  the  number 
of  students  to  be  trained.  This  conceptualization  of  the  training  process  fits  quite  well  with 
the  information  obtained  through  the  Interviews,  where  it  was  noted  that  the  job  demands,  yearly 
student  flow,  and  training  materials  were  the  major  factors  considered  in  course  design. 

Of  these  three  primal  variables,  only  occupational  difficulty  was  measured  directly. 
Manpower  requirements  and  subject-matter  difficulty  represent  latent  variables  defined  in  terms 
of  directly  measured  variables.  Subject-matter  difficulty  was  defined  by  the  diversity  of  course 
material,  the  reading  difficulty  of  training  materials,  expected  attrition  rates,  and  abstract 
knowledge  requirements.  Of  these  four  variables,  abstract  knowledge  requirements  exhibited  the 
largest  loading  for  subject-matter  difficulty,  suggesting  that  abstract  knowledge  requirements 
are  important  for  training  design  and  training  outcomes.  Manpower  requirements  were  defined  by 
selective  reenlistment  bonus  and  yearly  student  flow.  As  expected,  the  path  coefficients 
indicated  that  yearly  student  flow  was  the  strongest  index  of  manpower  requirements. 

Although  it  may  be  difficult  to  conceive  of  these  three  relatively  straightforward  variables 
driving  course  design,  this  hypothesis  was  strongly  supported  by  the  results  obtained  In  the 
present  effort.  For  instance,  these  three  variables  yielded  Rs  for  the  prediction  of  course 
length,  day  length,  student-faculty  ratio,  instructor  quality,  instructor  experience, 
instructional  support,  and  feedback  intensity  in  excess  of  .54  and  averaging  near  .60.  Thus, 
these  three  primal  variables  provide  an  excellent  definition  of  lower-order  course  variables. 
Moreover,  the  nature  of  the  paths  between  these  primal  variables  and  the  lower-order  course 
variables  was  readily  interpretable,  thus  supporting  the  validity  of  this  conceptualization. 

Two  latent  variables  were  among  the  lower-order  course  variables  held  to  be  driven  by  the 
three  primal  variables.  The  first  was  instructional  support,  which  was  defined  by  a  combination 
of  the  frequency  of  use  of  instructional  aids  and  the  number  of  hours  of  hands-on  Instruction. 
This  variable  reflects  the  frequency  of  use  and  the  amount  of  course  time  devoted  to  job  task 
simulations.  The  second  latent  variable,  feedback  intensity,  was  defined  by  the  amount  of 
feedback  and  practice  given  a  student.  The  relation  of  practice  to  feedback  intensity  may,  at 
first  glance,  appear  ambiguous.  However,  the  relation  is  based  on  the  fact  that  feedback  In 
basic  resident  technical  training  is  required  by  unit  of  instruction.  Thus,  less  time  per  unit 
equates  to  more  rapid  feedback. 

Within  the  model,  these  seven  course  content  variables  were  hypothesized  to  be  capable  of 
influencing  assessed  quality  of  performance  and  SIA  time  Independently  of  the  three  primal  course 
variables.  With  respect  to  the  magnitude  of  these  effects,  course  length.  Instructor  quality, 
and  feedback  intensity  had  the  greatest  impact  on  both  SIA  time  and  the  quality  of  student 
performance.  This  suggests  that  more  rapid  feedback,  a  better  overall  quality  of  Instruction, 
and  more  instructional  time  enhance  the  quality  of  performance.  On  the  other  hand,  the  paths 
obtained  for  instructor  experience,  day  length,  student-faculty  ratio,  and  instructional  support 


Mere  relatively  weak.  This  observation  should  not  be  taken  as  indicating  that  these  latter 
variables  are  irrelevant  to  student  performance.  Rather,  the  results  indicate  that  these 
variables  have  relatively  less  impact  on  training  outcomes  Mhen  the  influence  of  the  three  primal 
course  content  variables  is  partialed  out.  This  suggests  that  if  any  given  course  content 
variable  is  out  of  line  with  the  program's  status  on  course  subject-matter  difficulty, 
occupational  difficulty,  or  manpower  requirements,  it  might  have  a  marked  influence  on  the 
quality  of  student  performance. 

To  this  point,  little  attention  has  been  given  to  the  impact  of  the  three  primal  course 
content  variables  on  training  outcomes.  As  the  standardized  path  coefficients  indicate,  course 
subject-matter  difficulty,  occupational  difficulty,  and  manpower  requirements  all  have 
substantial  effects  on  both  assessed  quality  of  student  performance  and  SIA  time.  In  fact, 
course  subject-matter  difficulty  had  the  greatest  effect  on  SIA  time.  It  should  also  be  pointed 
out  that  the  primal  course  content  variables  generally  had  a  greater  Impact  on  assessed  quality 
of  student  performance  than  any  other  course  content  variable.  Thus,  it  appears  thrt  the  three 
primal  course  variables  not  only  drive  course  design,  but  also  constitute  those  course  content 
variables  most  likely  to  influence  student  training  performance. 

One  question  that  might  arise  at  this  point  concerns  the  relative  power  of  the  course  content 
and  student  input  variables  in  determining  assessed  quality  of  performance.  Overall,  evaluation 
of  the  standardized  path  coefficients  tends  to  indicate  that  the  course  content  and  student  input 
variables  make  roughly  comparable  contributions  to  the  prediction  of  assessed  quality  of  student 
performance,  with  the  student  input  variables  showing  a  slight  advantage.  This  is  not  an  unusual 
finding,  since  previous  research  (Tyler,  1964)  has  indicated  that  what  the  individual  brings  to  a 
course  has  i  more  important  influence  on  educational  outcomes  than  characteristics  of  the 
educational  system.  On  the  other  hand,  it  should  be  recognized  that  course  content  variables  did 
have  a  substantial  Impact  on  the  assessed  quality  of  performance.  Taken  as  a  whole,  the 
foregoing  considerations  indicate  that,  although  student  inputs  may  be  the  most  effective 
predictors  of  performance,  both  student  inputs  and  course  content  variables  should  be  considered 
in  order  to  attain  optimal  performance  prediction. 

Before  concluding  this  discussion,  one  additional  set  of  comnents  seems  in  order.  At  first 
glance,  the  path  coefficients  obtained  for  many  of  the  variables  examined  in  the  present 
investigation  may  seem  low  and  might  be  viewed  as  indicating  a  lack  of  effective  prediction. 
However,  the  size  of  the  Rs  generated  by  the  model  for  each  of  the  outcome  variables  was  quite 
large,  ranging  from  .50  for  nonacademic  attrition  to  .83  for  academic  attrition. 

If  one  grants  that  the  model  provides  a  stable  and  accurate  description  of  the  training 
process,  then  the  magnitude  of  the  individual  path  coefficients  can  be  viewed  In  a  different 
light.  Rather  than  diminishing  the  appropriateness  of  the  model,  they  suggest  that  no  single 
variable  or  limited  set  of  variables  Is  adequate  for  predicting  training  outcomes.  This 
observation,  in  turn,  confirms  the  initial  premise  of  the  current  investigation  that  resident 
technical  training  is  a  highly  complex  process  which  can  be  adequately  described  only  by 
considering  a  variety  of  student  input,  course  content,  and  training  outcome  variables.  Thus, 
the  model  developed  In  the  present  effort  Indicates  that  fully  adequate  predictions  of  training 
outcomes  can  be  obtained  only  when  training  is  conceived  of  as  being  a  complex,  multifaceted 
process  Involving  a  variety  of  student  inputs  and  course  content  variables.  Although  this  is  not 
an  especially  unusual  conclusion,  it  suggests  that  optimal  description  and  prediction  of  the 
training  process  can  never  be  attained  without  employing  multivariate  techniques  capable  of 
taking  into  account  these  complex  interrelationships.  Perhaps  much  of  the  value  of  the  present 
effort  lies  in  its  recognition  of  this  complexity,  and  its  use  of  a  heuristlcally  appropriate 
modeling  methodology  to  demonstrate  the  complexity  of  basic  resident  technical  training. 


Applications 

The  discussion  has  focused  primarily  on  the  empirical  model  of  resident  training  developed  In 
the  present  Investigation,  along  with  some  of  the  conclusions  that  flow  from  the  content  of  the 
model.  However,  since  the  model  was  developed  for  the  purpose  of  making  predictions  concerning 
the  impact  of  aptitude  requirement  adjustments  on  technical  training  outcomes.  It  is  necessary  to 
conclude  the  discussion  by  considering  the  potential  applications  of  the  model. 

Given  that  the  model  provided  both  an  accurate  description  of  the  Initial-skills  training 
process,  as  well  as  accurate  predictions  of  training  outcomes,  a  technical  constraint  that  might 
limit  application  of  the  model  must  be  examined.  To  employ  the  model  on  a  routine  basis  for 
predicting  training  outcomes,  it  will  be  necessary  to  specify  the  status  of  the  course  on  the 
various  student  input  and  course  content  variables.  As  a  result,  routine  application  of  the 
model  in  the  prediction  of  training  outcomes  requires  that  measures  of  student  input  and  course 
content  variables  are  readily  available.  Anticipation  of  this  constraint  resulted  in  the  use  of 
measures  of  student  input  and  course  content  variables  that  are  easily  obtained  from  sources  such 
as  the  PACE  data  file  and  standard  training  documents  such  as  the  P01,  training  plan,  and  other 
course  materials.  The  only  exception  is  the  ratings  of  course  abstract  knowledge  requirements 
made  by  the  ARRO  staff.  However,  given  the  simplicity  of  the  measurement  procedure  for  this 
variable,  there  would  be  little  difficulty  in  using  It  to  make  similar  ratings  when  the  need 
arises. 

The  foregoing  discussion  addressed  the  major  technical  issues  associated  with  application  of 
the  model.  Therefore,  it  would  now  seem  appropriate  to  turn  to  some  of  the  specific  applications 
of  the  model.  Assuming  the  availability  of  requisite  information  describing  student  input  and 
course  content  variables,  the  model  could  be  used  in  a  straightforward  fashion  to  predict  the 
effects  of  aptitude  requirement  adjustments  on  technical  training  outcomes.  For  a  specified 
course,  the  average  aptitude  of  individuals  assigned  to  the  associated  job  specialty  on  the  basis 
of  a  given  aptitude  requirement  minimum  would  be  obtained,  along  with  average  values  for  other 
student  inputs  and  values  which  describe  the  course's  status  on  course  content  variables.  These 
values  would  then  be  converted  to  standard  scores  and  used  with  a  series  of  mathematical 
equations  to  predict  average  values  on  the  training  outcome  variables.  These  equations  would 
produce  predictions  of  training  outcomes  in  standard  score  form.  To  facilitate  interpretation, 
it  would  be  desirable  to  convert  the  standard  score  predictions  back  to  raw-score  form.  The 
whole  student  input  model  could  be  manipulated  by  adjusting  the  means  of  certain  student  inputs 
to  obtain  training  outcomes  at  desired  levels. 

In  addition  to  the  applications  of  estimating  student  Inputs  required  for  entry  into  a 
training  program,  the  model  has  a  number  of  other  uses.  For  instance,  if  a  simulated  change  in 
aptitude  minimums  results  in  a  predicted  decrease  in  the  assessed  quality  of  student  performance, 
either  the  primal  course  content  variables  or  the  lower-order  course  variables  might  be 
manipulated  to  compensate  for  the  predicted  decrease  in  performance.  Hence,  the  model  m^y  be 
used  not  only  to  predict  the  effect  that  changing  student  inputs  has  on  training  outcomes,  but 
also  to  suggest  specific  training  strategies  for  compensating  for  negative  effects  of  these 
changes. 

In  addition  to  compensating  for  changes  in  student  inputs,  the  model  can  provide  a  framework 
for  course  design  by  evaluating  changes  in  course  subject-matter  difficulty,  occupational 
difficulty,  or  manpower  requirements.  Values  for  these  primal  course  variables  might  then  be 
used  to  predict  expected  values  for  lower-order  course  variables,  such  as  course  length  or 
student-faculty  ratio.  Predicted  values  for  these  latter  course  variables  as  they  follow  from 
changes  in  job  difficulty,  course  subject-matter  difficulty,  or  manpower  requirements  might 
provide  a  basis  for  decisions  concerning  course  design.  Moreover,  because  the  model  specifies 
the  relationships  of  the  primal  course  content  variables  and  the  lower-order  course  variables 
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with  the  various  training  outome  variables,  the  potential  impact  of  proposed  changes  in  course 
content  on  training  performance  can  be  predicted.  If  changes  in  course  content  result  in 
predictions  of  decreased  training  performance,  an  attempt  might  be  made  to  manipulate  other 
course  content  or  student  input  variables  so  as  to  avoid  such  a  decrease. 

Obviously,  the  model  has  many  potential  applications,  including  prediction  of  the  effects  of 
changes  in  aptitude  requirement  minimums  and  modifications  in  course  design  on  training 
outcomes.  However,  an  issue  that  is  likely  to  arise  in  such  applications  is  related  to  the 
multivariate  nature  of  training  outcomes.  In  the  refined  model,  there  are  five  training 
outcomes,  specified  as  academic  attrition,  nonacademic  attrition,  retraining  time,  academic 
counseling,  and  nonacademic  counseling,  all  flowing  from  the  assessed  quality  of  student 
performance  and  SI  A  time.  Thus,  in  evaluating  the  effects  of  any  simulated  change  in  student 
input  or  course  content  variables,  a  variety  of  potential  outcomes  must  be  considered.  Because 
multiple  outcomes  must  be  considered,  it  would  be  desirable  to  have  some  means  of  combining  these 
outcomes  into  a  single  decision-making  index.  One  way  of  accomplishing  this  is  to  estimate  the 
costs  associated  with  training  outcomes.  The  resulting  cost  estimates  would  then  provide  a 
conroon  framework  for  evaluating  changes  in  student  input  or  course  content.  Because  the  ATC 
Comptroller  maintains  training  cost  data  for  each  course,  it  may  be  possible  to  employ  these 
training  cost  data  in  the  implementation  of  the  model.  Therefore,  changes  in  training  outcomes 
could  be  linked  to  training  costs,  and  thereby  serve  as  a  basis  of  evaluating  the  overall  cost 
impact  of  student  input  or  course  content  adjustments. 

This  has  been  a  fairly  general  discussion  of  potential  applications  of  the  model  formulated 
in  the  present  effort.  This  discussion  has  underscored  the  point  that  the  model  will  not  only 
serve  the  intended  purpose  of  predicting  the  effect  of  aptitude  requirement  adjustments  on 
technical  training  outcomes,  but  can  also  serve  a  variety  of  other  purposes  as  well.  Perhaps 
more  important  than  these  relatively  pragmatic  applications,  the  study  may  have  made  one  other 
major  contribution.  By  defining  the  variables  relevant  to  technical  training  outcomes  and 
establishing  their  interrelationships,  the  model  provides  a  basis  for  facilitating  understanding 
of  the  initial-skills  training  process  as  a  whole. 

V.  CONCLUSIONS 

The  present  effort  was  undertaken  as  part  of  a  larger  Air  Force  effort  concerned  with  the 
enhancement  of  manpower  allocation.  More  specifically,  it  was  concerned  with  finding  some 

technique  for  assessing  the  impact  of  changes  in  aptitude  requirement  minimums  on  technical 
training  outcomes.  To  accomplish  this,  it  was  decided  that  an  empirical  model  should  be 
developed  by  which  student  input  and  course  content  variables  could  be  related  to  training 
outcomes. 

A  series  of  interviews  were  conducted  with  technical  training  instructors  and  administrators 
to  identify  the  student  input  and  course  content  variables  most  likely  to  influence  performance 
in  basic  resident  technical  training.  Readily  available  measures  of  these  variables  were  then 
specified.  Subsequently,  data  describing  the  status  of  some  5,000  trainees  in  39  courses  were 

obtained,  along  with  data  for  some  1,000  additional  trainees  in  nine  different  courses.  Path 
modeling  procedures  were  used  to  develop  an  empirical  model  of  the  technical  training  process. 

The  resulting  model  provided  an  effective  description  of  the  relationships  among  student 
input,  course  content,  and  training  outcome  variables.  The  observed  residual  (variance 

unaccounted  for)  was  only  .19,  and  the  model  yielded  an  R  of  .75  against  the  assessed  quality  of 

student  performance. 


A  number  of  interesting  facets  of  the  technical  training  process  were  identified  during  model 
development.  For  instance,  although  aptitude  was  one  of  the  best  predictors  of  the  quality  of 
student  performance,  it  interacted  with  a  variety  of  other  student  input  and  course  content 
variables  in  determining  training  outcomes.  Further,  it  appeared  that  course  content  variables 
tended  to  be  defined  by  course  subject-matter  difficulty,  occupational  difficulty,  and  manpower 
requirements.  Another  important  finding  was  that  finally,  the  model  indicated  that  counseling 
mediated  the  relationship  between  training  performance  and  distal  training  outcomes  such  as 
retraining  time  and  student  elimination. 

Results  of  the  analyses  indicated  that  the  model  could  be  used  both  to  predict  the  effects  of 
aptitude  requirement  adjustments  and  to  enhance  training  course  design.  Moreover,  it  appeared 
that  only  a  multivariate  modeling  approach,  such  as  that  employed  in  the  present  investigation, 
would  allow  effective  predictions  of  Air  Force  initial-skills  training  outcomes. 
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APPENDIX  A:  INTERVIEW  PROTOCOL 


Sample  of  Questions  Asked  Technical  Training 
Center  Staff  Members 


Course  Content 


What  determines  course  content? 

How  is  the  length  of  a  course  determined? 

How  much  Instructional  System  Development  (ISD)  effort  is  done? 

In  the  past  couple  of  years,  have  there  been  any  changes  in  the  courses  you  are  connected  with 
(e.g.,  changes  in  content  of  courses;  number  of  class  hours  per  day;  length  of  course;  entry 
aptitude  requirements)? 

How  are  these  changes  accomplished? 

What  has  been  the  effect  of  these  changes  on  student  performance,  amount  of  recycling,  attrition, 
quality  of  output,  etc.? 

If  you  encountered  changes  in  the  general  aptitude  level  of  your  students,  what  would  you  do  to 
preserve  the  quality  of  your  product? 

What  changes  would  you  like  to  see  made  in  your  courses  and  why  do  you  think  these  changes  would 
be  beneficial? 

How  do  student  aptitudes  affect  performance  in  your  courses?  Is  it  very  noticeable? 

What  changes  were  made  in  other  aspects  of  the  course  to  compensate  for  the  changes  in  aptitude? 

Have  there  been  any  changes  in  aptitude  requirements  to  your  courses?  Where  is  a  record  of  these 
changes  maintained? 

What  differences  have  you  noticed  between  non-prior-service  students  and  those  who  are  being 
retrained? 

If  no  changes  in  aptitude  requirements  have  been  made,  do  you  feel  that  changes  in  aptitude 
requirements  should  be  made?  Why? 

Besides  aptitude,  what  personal  characteristics  of  students  are  most  influential  on  their 
performance  in  training? 

How  do  students'  educational  experiences  affect  their  performance  (e.g.,  the  kind  of  courses 
taken)? 

Which  do  you  prefer:  group  lock-step  or  g^oup-paced  courses?  Why? 

How  should  the  difficulty  of  a  course  be  measured? 

Can  the  difficulty  of  a  block  of  instruction  be  estimated  from  Occupational  Survey  Report  (OSR) 
task  difficulty  ratings? 


What  aspects'of  a  course  have  the  most  effect  on  its  difficulty? 

How  do  you  get  feedback  about  your  courses? 

How  is  this  information  used? 

Is  it  possible  to  use  Career  Development  Course  (CDC)  completion  as  a  measure  of  graduates' 
quality? 

Student  Assessment 

How  is  student  performance  measured? 

What  are  the  differences  between  progress  checks,  performance  tests,  sampling  written  tests,  and 
comprehensive  written  tests? 

How  are  progress  checks  and  performance  tests  conducted? 

What  determines  which  kind  of  measurement  is  to  be  used? 

How  and  when  are  these  measurements  made? 

What  records  are  required  to  be  kept  on  student  performance? 

Are  there  any  important  aspects  of  student  performance  that  are  not  kept  on  the  ATC  Form  156? 

How  useful  is  the  ATC  Form  667  in  recording  progress  checks? 

How  much  consistency  is  there  across  instructors  in  grading  progress  checks  or  performance  tests? 
Have  you  ever  retested  students  on  block  tests  to  measure  reliability  of  tests? 

What  factors  influence  the  grading  of  progress  checks  or  performance  tests? 

How  are  these  assessments  combined  into  an  overall  assessment  of  student  performance? 

How  is  a  student's  final  school  grade  (FSG)  computed? 

What  is  the  usual  distribution  of  FSGs? 

Are  any  analyses  made  of  FSG  distributions? 

What  feedback  is  provided  as  a  result  of  these  analyses? 

How  are  distinguished  or  honor  graduates  chosen? 

What  is  the  difference  between  the  top  students  and  the  poorest  students? 

Where  and  how  do  these  differences  show  up? 

What  are  the  main  causes  of  poor  student  performance? 

How  well  informed  are  students  about  the  nature  of  the  work  they  will  be  doing  in  the  job 
specialty  for  which  they  are  training?  Does  this  affect  their  school  performance? 


Are  those  in  the  Guaranteed  Training  Enlistment  Program  (GTEP)  better  informed  or  motivated? 

Do  you  have  any  courses  where  the  nature  of  conditions  of  work  tends  to  create  stress  among  the 
students? 

What  inducements  can  lead  to  better  student  performance? 

How  do  you  determine  when  a  student  needs  Special  Individual  Assistance  { SIA) ? 

What  determines  how  much  SIA  is  given? 

Should  SIA  be  considered  as  additional  training  time  or  is  it  a  different  type  of  training? 

Where  is  the  information  concerning  SIA  recorded  and  how  accurate  is  it? 

Who  uses  the  SIA  information  and  how  is  it  used? 

What  determines  when  a  student  is  recycled  or  washed  back? 

Where  is  information  concerning  trainee  wash-back  recorded  and  how  is  it  used? 

What  determines  when  a  student  should  be  failed? 

Does  the  policy  toward  attrition  vary  over  the  months  or  years? 

What  are  the  reasons  for  attrition  and  how  do  you  determine  which  reasons  to  assign  to  a 
particular  case? 

What  factors  have  the  most  effect  on  the  various  categories  of  attrition? 

Course  Quality 

How  is  the  quality  of  a  course  measured? 

Where  may  information  concerning  course  quality  be  obtained? 

What  regulations  or  directives  govern  the  compiling  and  maintenance  of  course  quality  records? 
Which  are  better  indicators  of  course  quality? 

How  and  by  whom  is  Information  concerning  course  quality  used? 

How  far  back  do  records  concerning  course  quality  go? 

How  are  student  attrition  figures  computed? 

What  kind  of  feedback  concerning  course  quality  do  you  get  from  the  field? 

How  useful  are  student  critique  forms? 

Instructor 


What  effect  does  an  Instructor  have  on  how  much  a  student  learns? 


In  what  ways  do  instructors  influence  student  performance? 


m 


On  the  average,  how  many  instructors  teach  in  a  single  course? 
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How  is  the  required  number  of  instructors  determined? 

Are  there  times  when  there  aren't  as  many  instructors  available  as  are  required? 
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ATC  Form  281  is  used  to  evaluate  instructors.  Of  the  aspects  measured  by  that  form,  which  are 
the  most  important?  Which  the  least  important? 

How  should  we  measure  instructor  quality? 

How  does  an  instructor's  field  experience  or  lack  thereof  affect  his/her  performance? 

What  are  the  characteristics  of  good  and  poor  instructors? 


m 

r  ***? 

r,Vi 


<  .*] 


i  .’iii 


m 


$ 


APPENDIX  B:  STUOENT  MEASUREMENT  GUIDELINES 


DEPARTMENT  OF  THE  AIR  FORCE 
Headquarters  Air  Training  Command 
Randolph  Air  Force  Base,  Texas  78150 


ATCR  52-3 


25  July  1982 


Technical  Training 
STUDENT  MEASUREMENT 

This  regulation  establishes  policy,  assigns  responsibilities,  prescribes  procedures,  and  provides  guidance  for  student 
measurement  in  technical  and  military  training  programs  consistent  with  AFM  50-2.  it  applies  to  all  technical  training 
wings.  3785FLDTG.  School  of  Health  Care  Sciences.  3480TCHTG.  and  the  Basic  Military  Training  School 


1.  Terms  Explained: 

a  Appraisals.  A  group  of  questions  and  (or)  projects 
used  to  check  the  day-to-day  learning  process  Appraisals 
are  an  informal,  optional  measurement  and  are  not  a  part 
of  the  recorded  measurement  program 
b  Comprehensive  Written  Test.  Written  test  items 
used  to  lully  measure  student  achievement  ol  a  knowledge 
oriented  objective(s)  not  included  on  progress  checks. 

c.  Compromise  of  Test  Materials.  Any  act  or 
occurrence  whereby  students  gain  unauthorized  knowledge 
of  contents  of  test  materials. 

d.  Criterion  Checklist.  A  reference  or  listing  on  A  1C 
Forms  667  or  667A.  Criterion  Checklist,  of  course 
objectives  that  are  designed  to  be  assessed  by  progress 
checks.  In  courses  without  a  I’Ol.  objectives  are  extracted 
from  lesson  plans  and  listed  on  the  criterion  checklist. 

e  Criterion  Referenced  Testing  (CRT),  lest  to 
determine  if  the  behavior  specified  in  the  objective  has  been 
acquired  May  involve  multiple-choice,  completion,  true  or 
lalxe.  matching,  essay  items,  short  answer  or  actual 
performance  of  a  task. 

f.  End-of-Course  Test.  Written  test  items 
administered  near  the  end  of  a  course  to  cheek  the  retention 
of  the  knowledge  component  of  selected  objectives  an 
optional  quality  control  check. 

g  Master  Copy  of  Written  Test.  A  copy  of  each 
written  test  annotated  to  ( I)  identify  the  correct  answers,  (2) 
show  POI  objective  reference  which  each  item  supports, 
and  (3)  list  the  approved  corrections  and  changes  made  in 
the  test.  It  must  also  contain  a  statement  showing  approval 
for  continued  use  when  the  test  date  is  earlier  than  the  POI 
date,  or  the  course  chart  date  for  courses  that  have  no  POI 
h.  Measurement  Plan.  A  plan  designating  the  method 
uved  to  measure  student  achievement  ol  the  course 
objectives  and  indicating  the  correlation  between  test  items. 
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and  POI  objectives  and  the  correlation  heween  PO 
objectives  and  training  standard  items 

i.  Objective.  A  precise  statement  of  the  student' 
behavior  to  be  exhibited,  the  minimum  standard  o 
performance  or  proficiency  expected  and  the  condition 
under  which  the  behavior  is  to  be  exhibited  (AFM  50-2 
attachment  I). 

(1)  Knowledge  Oriented  Objective.  An  objects 
originating  on  the  basis  of  a  training  standard  dementis 
requiring  task  or  subject  knowledge.  Conditions  am 
standards  may  be  implied  if  they  are  clearly  obvious. 

(2)  Performance  Oriented  Objective.  An  objectiv 
originated  on  the  basis  of  a  training  standard  dementis 
task  performance.  Standards  may  be  implied  if  clear) 
obvious. 

j.  Performance  Exercise.  The  means  of  providing  th 
students  practice  sessions  throughout  the  learning  period 

k.  Performance  Test.  An  instrument  administered  in 
test  mode  and  environment  after  the  teaching-learnm 
activity  to  verify  student  attainment  of  pcrformanc 
oriented  objectives. 

l.  Performance  Recheck.  A  rceheck  of  studen 
performance  on  a  specific  objective,  normally  administered 
near  the  end  of  the  course  It  is  mandatory  when  a  specia 
requirement  has  been  identified  in  the  training  standard  o 
where  a  separate  agency  certification  is  required.  It  i 
optional  when  objectives  require  added  quality  control. 

m.  Prograss  Check.  The  assessment  of  student  ai 
complishment  of  knowledge  or  performance  objective 
during  the  teaching-learning  activity.  Results  ar 
documented  on  ATC  Form  667  or  667A 
n  Progress  Checklist.  (ATC  Form  98  or  locall 
approved  checkhs!)  The  breakdown  of  an  objective  int 
elements  or  steps  necessary  to  determine  whether  eac 
student  satisfactorily  accomplishes  an  objective  during 
progress  check  or  performance  test. 

o  Sampling  Written  Test.  Instrument  used  to  sampi 
the  knowledge  components  of  each  course  objectiv 
assessed  during  a  progress  check 

p  Test  Item  Pool.  A  file  of  written  test  items  which  ea 
be  used  to  construct  or  revise  a  test  Items  may*-  b 
handwritten,  be  a  copy  of  a  test  replaced  by  a  newe 
revision,  or  a  copy  of  a  test  containing  annotated  propose, 
changes  (working  copy).  Maintenance  ol  a  lest  uem.pool  i 
optional.  • 

q.  Test  Materials,  tests  or  page's  from  a  test.  tc> 
booklets,  scoring  keys,  test  items  in  pools,  complete. 
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answer  sheets,  test  data  forms,  automatic  data  processing 
cards  as  well  as  calculations,  diagiams.  formulas,  etc.,  used 
b>  a  student  while  being  tested. 

r.  Training  Deficiency.  A  condition  that  prevents 
accomplishment  of  training  requirement!')  and  which 
cannot  be  corrected  before  the  affected  students  graduate. 


2.  Student  Measurement  Concept.  The  ATC  technical 
training  student  measuicmcnt  program  is  predicated  on 
valid,  job  relevant,  and  properly  constructed  objectives. 
Since  the  objectives  specify  precisely  what  behavior  is  to  he 
exhibited,  the  condition  under  which  the  behavior  will  he 
accomplished,  and  state  or  imp!)  the  minimum  standard  of 
acceptable  behavior,  they  are  the  basis  for  measurement  of 
student  achievement  finch  ciitcrion  test  item  is  based  solely 
on  the  requirements  specified  in  the  objective  which  it  is  to 
tneasuic  This  program  normally  includes  instructor 
assessment  ol  each  student's  accomplishment  of  each 
objective  during  the  teaching-learning  environment 
thmugh  the  use  of  progress  checks  Progress  checks  aie  the 
most  practical  means  of  verifying  accomplishment  of  each 
course  objective  However,  when  it  is  more  appropriate  to 
delay  the  measurement  of  certain  objectives  and  satisfy 
them  through  ihc  use  ol  comprehensive  written  and  (or! 
ivitoiin.incc  tests,  stuietiue  the  measurement  device  in  a 
manner  which  dctei mines  students'  achievement  of  each 
objective  tested  1  he  objectives  assessed  by  a  progress  check 
during  the  teaching-learning  activity  are  sampled  on  unit, 
block,  and  (or)  end-of-coursc  written  tests  to  aid  in 
retention  and  act  as  quality  control  devices.  Additionally,  it 
's  olien  appropriate  to  rccheck  certain  performance 
oriented  objectives  with  performance  rechccks  Normative 
grading  systems  (grading  on  a  curve)  aie  not  used 


3.  Objectives  of  Student  Measurem^nt: 

a  linsine  each  student  achieves  every  course  obicctivc 
hclorc  graduation. 

b.  Permit  early  identification  of  students  who  fail  to 
attain  objectives  and  need  special  individual  assistance  or 
additional  training 

c  Inform  students  whether  or  not  they  aie  attaining  the 
ohiectivcs.  stimulate  effective  learning;  and  reinforce 
knowledges  and  skills 

d  Fnsurc  students  meet  the  perlomiance  rccheck 
requirements,  such  as  typing  or  code  speed  specilicd  by 
using  activities,  or  whcie  a  separate  agency  ccriilicaimn  is 
required 

t  PtoviJc  data  for  use  in 

<l)  Determining  teaching  and  learning  cHcciocncss 
and  improvement  of  the  instructional  system 
(2 1  Ma  intainmg  quality  control 

(t)  f  stablishmg  a  permanent  icroid  ol  each  student's 
achievement  in  the  course 

(4)  Selecting  students  loi  special  recognition 
( A )  Validating  aptitude  lest  batteries 


4.  Policies: 

a  Before  graduation,  each  student  achieves  all  course 
objectives  as  demonstrated  by  a  progress  check,  com¬ 
prehensive  written  test,  or  performance  test  and  pass  all 
sampling  written  unit,  block,  and  (or)  cnd-of-cour>e  tests 
Students  may  graduate  it  failure  to  accomplish  objectives  is 
due  to  the  existence  ol  a  training  deficiency  Document 
training  deficiencies  as  icquired  in  Af-R  H-13  ATCSup  I 
h  Student  measurement  is  not  required  in  orientation, 
symposium,  and  tamih.iri/ation  type  courses  Aw.ud  final 
course  grade  of  satisfactory  (S) 

c  Use  student  measurement  in  all  type  2,  3.  and  4 
courses  (AFM  50-5)  and  military  training  courses  except 
those  in  h.  above. 

d  Measurement  ol  students  attending  type  I  and  5 
courses  follows  the  pioecdures  specified  in  the  contract  jnd 
appropriate  course  documcntion  or  as  specified  by  the 
agency  responsible  lor  establishing  the  curriculum  for 
training.  Where  possible,  measurement  follows  the  intent  of 
this  regulation 

e.  The  pioccduics  lor  airman  basic  military  training 
courses: 

(1)  Record  w rnteii  test  and  performance  test  grades  on 
locally  designed  hum. 

(2)  Record  performance  test  grades  as  satisfactory  ( S ) 
or  unsatisfactory  (ll 

(3)  1  sc  adieetival  grades  of  outstanding  (Ol.  satisfac¬ 
tory  (S).  or  unsatisfactory  (U)  for  written  tests 

(4 )  Record  overall  grades  as  outstanding  (O).  satisfac¬ 
tory  (S).  or  unsatisfactory  (III 

I  lime  allotted  lor  administration  ol  written  tests,  test 
critiques,  and  performance  tests  is  shown  in  the  course 
chart  and  POI.  The  time  allotted  must  allow  the 
measurement  of  each  student  on  each  objective,  tsiablish 
special  policies  and  procedures  consonant  with  AFM  50-29. 
as  appropriate,  for  testing  Security  Assistance  1  raining 
Program  trainees.  Frequent  measurement  is  encouraged  to 
mmimi.’c  washhack  time  and  to  prnv  ide  carlv  identification 
of  students  in  need  of  special  individual  assistance 

g.  Develop  sampling  written  unit,  block,  and  (or)  end-of- 
course  tests  to  sample  the  knowledge  components  ol  each 
objective  assessed  during  a  piogress  cheek  I  lie  primary 
purposes  of  these  tests  are  to  aid  in  retention,  act  as  a 
quality  control  device,  and  assign  grades  L'laMish  a  p.i" 
or  tail  point  no  lower  than  Ml  percent 

h  When  progress  cheeks  are  not  used  to  assess  situUv.! 
accomplishment  of  knowledge  oriented  ohiectivcs.  develop 
comprehensive  wiittcn  tests  to  determine  student  achieve¬ 
ment  1  he  pass  or  fail  point  is  established  by  the  'tandnrj 
stated  in  c.ich  obicctivc. 

i  (trades  aie  assigned  for  both  written  and  pei  lot  maikc 
tests  (ir.ide  vvnncn  tests  using  percentage  seoics  1‘erloi- 
manee  tests  may  he  graded  using  either  the  satisfactory  I S I 
oi  unsatistaetory(T)  method,  oi  percentage  scores  When 
the  S-l  method  is  used  on  performance  tests,  i he  final 
course  giade  is  the  average  of  peieeiitage  semes  attained  on 
written  test'  When  a  percentage  score  i-  used  on 
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performance  tests,  the  final  course  grade  is  the  average  of 
the  scores  obtained  on  the  performance  and  written  tests 
j  Establish  and  maintain  a  documented  course 
measurement  plan  indicating  the  measurement  device  used 
to  satisfy  each  objective. 

5.  Responsibilities: 

a  ATC  TT  SA  establishes  basic  policy.  provides 
guidance,  and  maintains  surveillance  over  the  student 
measurement  program 

b  Training  wings  schools  3785FLDTG  provide  staff 
supervision  and  measurement  expertise  to  ensure  that 
student  measurement  as  prescribed  herein  is  implemented 
and  remains  consonant  with  the  intent  of  this  regulation. 

c  Groups  and  departments  implement  and  supervise  the 
student  measurement  program  according  to  this  regulation 
by: 

(1)  Establishing  the  passing  standard  for  each  sampl¬ 
ing  written  test  Do  not  set  the  minimum  grade  below  the 
equivalent  of  60  percent. 

(2)  Approving  progress  checks,  written  and  perfor¬ 
mance  tests,  and  related  materials  and  instructions 
including  rev isionv 

(.1)  Authorizing  establishment  and  retention  of  test 
item  pools. 

(4)  Ensuring  that  instructors  and  instructor  super¬ 
visors  are  knowledgeable  of  the  procedures  for  conducting 
the  measurement  program  for  their  respective  courses. 

(5)  Providing  guidance  and  measurement  expertise  to 
individuals  involved  in  preparing  draft  tests  and  test 
materials,  administering  and  critiquing  progress  checks  and 
tests,  and  documenting  measurement  results. 

(6)  Conducting  test  analysis  and  revising  the  measure¬ 
ment  program  as  required 

6.  General  Procedures: 
a  Measurement  Plan: 

(1)  For  each  course  objective,  document  the  type  of 
measurement  dev  ice  (performance  test,  progress  check,  or 
comprehensive  written  test)  used  to  satisfy  each  objective, 
the  test  number,  the  specific  measurement  ilein(s)  covering 
each  objective,  and  the  training  standard  item  supported  by 
each  ohicctive 

(2)  I  he  measurement  plan  may  be  documented  on  the 
POf.  part  I  of  the  lesson  plan,  or  on  a  separate  document 

b  Progress  Checks: 

(1)  l  ist  oi  reference  the  objectives  on  \l(  Form  66? 
or  667A  that  are  to  be  assessed  through  the  use  of  a 
progress  check 

(2)  Fach  student's  performance  is  evaluated  hv  the 
instructor  on  each  objective  that  is  to  be  satisfied  hv  a 
progress  check  while  the  student  is  in  the  teaching-learning 
environment  The  instructor  determine-  if  the  student 
successfully  accomplished  the  objective  based  on  the 
beh.ivior.il  outcome  stated  in  the  ob|eetive  applicable 
checklist  (AT  C  Form  98  or  locally  produced  checklist),  and 
his  or  her  tudgment  of  the  student's  performance  For 


objectives  that  require  group  activity,  the  instructor  rate- 
each  member  according  to  the  individual's  performance 
and  participation  in  the  group. 

(.1)  When  a  progress  check  cannot  be  conducted 
during  the  block  because  of  equipment  malfunction, 
nonavailability,  etc,  the  student  may  continue  in  training 
provided  the  progress  check  or  a  performance  test  on  the 
same  objective  is  completed  before  graduation.  When  the 
equipment  malfunction,  nonavailability,  etc.  cannot  hi 
corrected  before  graduation,  document  this  as  a  training 
deficiency.  See  APR  8-13  ATC  Sup  I. 

(4)  After  the  instructor  determines  that  the  student  ha- 
satisfactorily  attained  an  objective  identified  on  the 
criterion  checklist,  an  “S"  or  percentage  score  is  entered  on 
ATC  Form  667  or  667A  Enter  “U"  for  unsatisfactorv 

(5)  If.  alter  special  individual  assistance  or  practice,  a 
student  cannot  attain  a  satisfactory  rating  on  a  progre— 
ch  ck,  then  consider  washback  or  removal  from  training 
action 

(6)  Retain  completed  ATC  Form  667  or  667A  until 
the  results  are  recorded  on  ATC  Form  156.  Student  Recoid 
of  Training  Retain  the  ATC  Form  667  or667A  completed 
during  validation  until  the  next  course  revision  Dcstrov  n 
according  to  AFM  12-50,  table  50-1.  rule  12 

(7)  Record  the  overall  rating  for  the  block  as  an  “S." 
"I1.”  or  percentage  score  of  the  ATC  Form  156  or  379 
Attendance  and  Rating  Record  If  a  student  is  removal 
from  training,  annotate  the  ATC  Form  156  or  379  a- 
prescribed  in  ATCR  52-11. 

c  Performance  Tests: 

(1) Test  each  student  on  every  objective  that  i- 
designated  for  a  performance  test.  When  the  objective  call- 
tor  group  performance,  the  instructor  rates  each  mcmhci 
according  to  the  indiv  idual's  performance  and  participation 
in  the  group. 

(2)  Develop  a  checklist  for  each  objective  measured  by 
a  performance  test  using  ATC  Form  98  or  locally  approved 
checklist  The  checklist  should  include  specified  standard- 
relative  to  instructor  assistance  and  safety,  a  breakdown  oi 
technical  data,  an  explanation  of  its  use.  or  any  othe- 
intonnation  necessary  to  ensure  a  high  degree  of  ohicctiv  it y 
in  determining  satisfactory  performance. 

(3)  Develop  standardized  instructions  to  (he examine' 
for  administering  and  critiquing  the  performance  test  Sc. 
attachment  3 

(4)  When  practical,  an  instructor  other  than  tin 
individual  who  taught  the  class  should  administet  the  test 

(5)  Inform  the  students  of  their  grades  and  critiqu- 
their  performance  as  soon  as  practical  after  administration 

(6)  Record  performance  lest  grades  for  each  ohicctiv 
(performance  test  item)  as  "S."  *'U."  or  percentage  score  ot 
A  1C  Form  98  or  locally  developed  checklist 

(7)  Record  the  overall  rating  for  the  performance  tc-' 
a-  “S."  "I  or  percentage  score  on  the  AlC  Form  15'-  o 
379  A  satisfactory  performance  test  rating  require 
successful  accomplishment  of  each  objective  mea-ured  a: 
this  test  point  in  the  course 
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(8)  Reijm  ihe  completed  ATC  Form  98  oi  loc.ilK 
approved  form  until  results  are  entered  on  Alt'  Form  156 
or  379  Retain  the  ATC  Form  98  accomplished  during 
validation  until  the  next  course  revision 

(9)  Students  w  ho  fail  performance  tests  arc  considered 
for  special  individual  assistance,  uashback.  or  removal 
from  training  If  a  student  is  removed  from  training, 
annotate  ihe  Alt'  Form  156  or  379  as  prescribed  in  A  ICR 
52-11 

d  Written  Tests: 

(1)  Develop  a  sampling  written  test  for  those  objec¬ 
tives  assessed  by  progress  checks.  The  sampling  should 
include  ever)  objective  when  all  test  versions  are  combined. 
No  single  lest  version  must  include  ever)  objective  Propel 
rotation  ol  test  versions  includes  all  obiectives  over  a  period 
ol  time  and  permits  validation  of  each  test  item.  Ihe 
dillicully.  complexity,  criticality,  and  scope  of  behavior 
spcciticd  by  the  objective  is  used  in  determining  the  number 
of  test  questions  required  lor  each  objective 

(2)  II  the  knowledge  objectives  have  not  been  satisfied 
through  the  use  of  progress  checks,  construct  a  comprehen¬ 
sive  written  test  so  as  to  provide  for  documentation  of  the 
attainment  of  the  objectives  tested  The  number  of 
questions  relating  to  a  particular  objective  is  determined  by 
the  objective  itself.  For  example,  if  an  objective  includes  a 
standard  of  "4  out  of  5."  the  test  must  have  five  questions 
on  that  ob|cctivc  and  the  student  must  get  at  least  loin  ol 
these  questions  correct  to  demonstrate  satisfactory  achieve¬ 
ment  of  the  objective 

(3)  Develop  enough  test  versions  (minimum  of  two) 
for  each  measurement  point  in  a  course  to  pi  ov  idc  alternate 
tests  in  the  event  ol  a  test  compromise  1  he  selection  ol 
questions  should  not  exceed  25  percent  ol  the  questions  in 
any  other  test  version  Do  not  count  scrambled  or 
resequenced  items  as  test  version 

(4)  Dev  clop  standardized  instructions  to  the  examiner 
for  administering  and  critiquing  the  written  test  See 
attachment  4 

(5)  I  nsupervised  breaks  are  not  taken  during  the 
administration  ol  written  tests  Students  may  leave  the 
room  on1)  when  they  have  completed  then  lest  and  on 
submission  ol  then  testing  m.itenals  to  the  examiner 

1 6 1  I  se  technical  data  whenever  it  contributes  to  on- 
the-job  realism 

(')  Tests  are  critiqued  and  students  informed  ol  their 
grade  as  soon  as  practical  alter  administration  Scored 
answer  sheets  and  copies  ol  the  lest  are  letuiued 
temporarily  to  all  students  for  their  review  during  the 
critique  (except  BM  I  i  lake  care  to  reduce  the  possibility 
of  test  compromise 

(8)  Record  written  lest  results  on  Alt  I  min  1 56  or 
3^9  as  a  percentage  score  Circle  failing  semes 

(9)  When  a  studenl  receives  a  failing  grade,  lollow  the 
procedures  in  ATCR  52-26.  paragraph  Ik  W  hen  a  student 
passes  the  retest,  record  the  minimum  passing  sc  me  allowed 
by  the  test  as  that  student's  grade  and  indicate  in  the 
remarks  section  ol  the  A  I  (  term  1 5<>  the  student''  actual 


score.  Remove  a  student  from  training  when  if  becomes 
apparent  the  student  cannot  complete  the  course. 

(10)  Conduct  an  analysis  of  test  items  as  soon  as 
possible  alter  each  test  for  the  first  three  classes.  200  cases, 
or  I  vcar.  whichever  is  sooner.  After  initial  validation, 
conduct  further  test  analysis  on  a  random  basis.  The  intent 
of  lest  item  analysis  is  to  ensure  quality  control  and 
determine  the  effectiveness  of  instruction,  course  materials, 
and  test  items.  Place  emphasis  on  "high"  and  "low1  ("high" 
miss  50  percent  or  more  of  students  miss  a  question  and 
“low"  miss  less  than  5  percent  of  students  miss  a  question) 
miss  rates  to  determine  what  corrective  action,  if  any.  is 
required,  l  ook  for  tiends  of  "high"  and  “low"  miss  items 
over  an  extended  period  of  lime.  Document  lest  item 
analvsts  on  Al  t'  Form  068.  Test  Data,  (see  attachment  I ). 
a  locallv  produced  lorm.  or  a  computer  product.  Maintain 
test  analysis  until  a  test  is  revised,  deleted,  or  would  serve 
no  further  purpose. 

c  End-of-Course  Written  Tetl /Performance 
Recheek: 

ll)  Construct  end-ol-course  tests  to  check  selected 
knowledge  oriented  objectives  for  courses  where  such  a  test 
is  necessary  or  desirable  Performance  rechecks  are 
administered  wlicic  a  special  icquirement  such  as  a  typing 
or  code  speed  is  included  in  the  course,  where  separate 
agenev  certification  is  required,  or  when  objectives  require 
added  quality  conttol  Other  objectives  may  be  checked  on 
a  sampling  basis  as  deemed  appropriate  The  amount  of 
time  devoted  lor  end-ol-course  testing  and  performance 
rcchecks  should  be  consistent  with  the  need  for  such  testing 
and  be  cost  effective. 

(2)  Each  student  is  adniinisteied  a  performance 
rccheck  on  each  objective  developed  lor  special  re¬ 
quirements  and  certification 

(3)  Concrete  terms  (such  as  word'  pei  minute,  etc. I 
used  loi  test  grades  ate  recorded  in  A1  C  Form  156.  item  3. 
part  Y. 

(4)  \k  hen  cnd-ol-coursc  tests  arc  used,  each  student 
must  obtain  a  passing  score  on  (he  written  lest  to  be 
graduated  lest  glades  are  recorded  on  A  1C  Form  156  01 
379 

(5)  Students  who  tail  to  xatixlaclorily  complete 
perlorm.incc  rcchecks  01  do  not  meet  ceitilication  01 
special  requirements  will  not  be  graduated 

1  Appraisals: 

ll)  l  sc  appraisal'  as  an  interim  check  lor  ictcntion 
and  cailv  identification  ol  students  who  need  special 
individual  assistance 

(2)  Kecoidmg  01  maintaining  grades  and  control  0 
not  requited 

(3)  Appiaisals  111.1v  be  administered  wit  bout  doe  11  men¬ 
tation  in  tile  I’OI 

7.  Control  of  Testing  Materials. 

a  Stoic  lest  materials  in  a  storage  cabinet  secured  w  if h  a 
combination  lock,  in  a  sjlc.  or  othci  computable  secure 
aica  I  c'l  inatcnals  mainlaincd  in  wold  processing  cenlcix 
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on  tapes  or  floppy  discs.  and  those  ill  diall  stages  will  be 
secured  in  the  same  manner  as  a  linalired  test;  however,  the 
provisions  of  paragraphs  7c  and  d  below  do  not  apply 
Control  performance  tests  and  materials  only  when  they 
contain  information  that  could  cause  a  compromise  ol  the 
test 

b  Restrict  access  to  materials  in  locked  storage  to  the 
minimum  number  of  authored  personnel  necessary. 

c  Maintain  A  1C  f  orm  1005.  l  est  Control,  when  a  test 
is  removed  from  a  locked  storage  area  for  any  purpose,  and 
ensure  pertinent  details  arc  entered  in  the  "issued  to 
columns  The  A  I  C'  Form  1005  is  not  a  controlled  form  and 
need  not  be  stoied  in  a  secured  area. 

d  Alter  a  test  is  administered,  the  test  administrator 
examines  the  test  booklets  for  missing  pages,  removes 
pencil  marks,  and  returns  booklets  to  test  custodian.  The 
lest  administrator  certifies  to  the  reusability  of  the  tests  by 
signing  the  test  control  form.  Custodians  signature 
indicates  all  issued  tests  have  been  returned  and  the  current 
inventory  count  as  shown  on  ATC  Form  1004.  Test 
Inventory,  is  accurate.  After  the  last  entry,  destroy  ATC 
Forms  1005  immediately  according  to  ATM  12-50.  table 
II  I.  rule  4 

c  I  .'sc  a  separate  A  I  C  Form  1004  to  maintain  a  running 
inventory  ol  each  different  controlled  test  and  its  related 
test  materials.  Keep  reserve  stocks  of  controlled  tests  to  a 
minimum.  Fxcept  lor  course  validation  file  copy,  destroy 
outdated  controlled  tests  and  related  test  materials  within 
10  duty  days  after  implementation  date  of  the  replacement 
test  Immediately  destroy  loose  pages  lelt  over  alter 
controlled  tests  are  assembled  or  maintain  inventory  of 
loose  pages  on  ATC  Form  1004  Immediately  destroy 
closed  out  A  1C  Forms  1004  according  to  AFV1  12-50.  table 
I  I  - 1 .  rule  4 

I.  Suspend  use  ol  a  compromised  test  or  one  suspected 
of  being  compromised  for  a  period  determined  by  training 
development  sections  or  measurement  sections, 
e  Complete  ATC  Form  1004  as  follows: 

(1)  Type  or  make  all  entries  in  ink 

(2)  I  he  master  copy  of  the  written  test  is  not  included 
in  the  balance  on  hand  column  The  master  copy  is 
identified  by  writing  across  the  top  center  of  the  first  page 
"Master  Copy"  with  no  numerical  or  alpha  idcntilier 

(3)  Make  a  separate  and  complete  entry  for  each  type 
of  material  used  Where  necessary,  use  more  than  one  line 
in  the  idcntilication  ol  material  and  action  taken  column 

(4)  Custodians  Block  Rnlcr  statement.  "Posted  on 
storage  cabinet 

(5)  Idcntilication  of  lest  Muteiials  l  sc  the  following 
code  to  identity  miscellaneous  test  materials 

(at  K  test  key ( s >. 


(b)  I)  test  data  loim(s) 

< c )  Additional  coding  at  user  discretion 

(6)  Transaction  Number  Each  individual  transactio 
is  numbered  in  sequence. 

(7)  Balance  on  Fland.  After  each  transaction 
completed,  enter  the  new  balance  in  the  column  provided 
If  one  series  of  tests  is  destroyed  and  the  same  ATC  Fon 
1004  is  used  for  the  subsequent  series  of  tests,  start  th 
transaction  number  over  at  I. 

h.  Destroy  completed  answer  sheets  for  written  test 
immediately  after  results  have  been  recorded  and  test  itci: 
analysis  is  completed,  unless  required  for  specific  projects 
in  this  case  maintain  an  inventory  on  a  general  purpos 
form. 

i.  Control  classified  tests  under  the  above  provisions  i 
addition  to  those  safeguards  outlined  in  AFR  205-1  o 
other  applicable  security  directives. 

8.  References: 

a  AFR  8-13' ATC  Sup  I.  Air  Force  Specialty  Trainin 
Standards. 

b.  AI  M  12-50.  Disposition  of  Air  Force  Document. i 
tion 

c  AFM  50-2.  Instructional  System  Development. 

d.  AFM  50-5.  IhSAF  Formal  Schools  Catalog. 

e.  AFM  50-62,  Principles  and  Techniques  of  Instruc¬ 
tion 

f.  AFP  50-58.  Handbook  for  Designers  of  Instruction;! 
Systems,  volume  III 

g.  A  ICR  50-21.  Field  training  Detachment  (FID 
Program 

h  ATCR  52-6,  Curricula  Documentation, 
i.  ATCR  52-11.  Student  Record  of  Training. 

I  ATCR  52-26.  Student  Scheduling  and  Administr.i 
turn 

k  ATCR  53-1.  Faculty  Boards 
I  A1CM  52-9.  Guide  lor  Special  Training 
in  AFR  205-1.  Information  Security  Program 

9.  Supplements.  Each  center  and  school  DPI 
coordinates  supplements  to  this  regulation  with  Ht 
A1C  TTSA  for  approval  before  publication. 

10.  Implementation.  Revise  each  course  to  comply  wit 
the  provisions  of  this  regulation  during  the  next  cours 
revision  or  within  I  year  of  the  dale  of  this  publicatioi 
whichever  is  sooner 

11.  Forms: 

a  Prescribed.  ATC  98.  667  667A.  668,  1005,  and  1004 
b  Adopted.  AfC  156  and  379 
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Director  of  Administration 


THOMAS  M.  RYAN.  JR..  General.  USA! 
C  ommander 


4  Attachments 

1.  Completion  Instructions  for  ATC  Form  668.  Test  Data 

2.  Completion  Instructions  for  ATC  forms  667  and  667a 

3.  Instructions  to  the  Examiner — Performance  Test 

4.  Instructions  to  the  Examiner — Written  Test 


SUMMARY  OF  CHANGES 

This  regulation  clarifies  guidance  concerning  written  and  performance  tests;  differentiates  between  knowledge  and 
performance  objectives;  authorizes  the  use  of  progress  checks  to  satisfy  objectives,  establishes  the  use  of  the  ATC  Form  668 
as  optional  for  test  analysis;  authorizes  the  use  of  a  sampling  written  block  tests  and  end-of-course  tests  as  retention  devices; 
adds  the  requirement  to  list  all  objectives  assessed  by  progress  checks  on  the  ATC  Form  667  or  667A:  authorizes  the 
conversion  of  performance  test  results  to  a  percentage  score  and  count  as  part  of  the  overall  course  grade;  adds  the 
requirement  to  have  a  documented  measurement  plan;  clarifies  the  test  control  procedures  for  tests  stored  in  word 
processing  centers  on  tapes  or  floppy  discs,  establishes  the  requirement  to  have  supplements  to  this  regulation  approved  by 
t  is  headquarters  before  publication;  and  establishes  implementation  procedures  for  complying  with  the  provisions  of  this 
regulation 
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COMPLETION  INSTRUCTIONS  FOR  ATC  FORM  668,  TEST  DATA 


The  numbers  below  refer  to  corresponding  circled 
numbers  in  specific  locations  on  ATC  Form  668  shown 
in  figure  A  I  - 1 . 

1.  Enter  class  number  and  section  or  shift  of  the  group 
tested  Example:  68121 1  A.  For  self-paced  courses,  enter 
date  test  data  was  computed. 

2.  Enter  the  number  of  students  in  the  group  tested  Do 
not  count  or  use  any  answer  sheets  of  retests  or  foreign 
students.  Example:  8. 

3.  Enter  the  last  name  of  instructor  that  taught  the  group. 
Example:  Brown.  In  self-paced  or  multigroup  entry  courses 
lease  blank. 

4.  Enter  the  course  number.  Example:  3ABRXXXX. 

5.  Enter  number  of  this  specific  test.  Example:  W3I2-I. 

6.  When  the  form  has  the  required  number  of  groups 
recorded  in  sections  I  through  5.  enter  the  total  number  of 
students  recorded  in  sections  I  through  5  in  this  space. 
Example:  42. 

7.  Draw  a  heavy  line  through  or  block  out  the  correct 
answer  for  each  question  on  the  test 

8.  Using  the  answer  sheets  (ATC  Form  26  or  26a)  record 
the  tot answers  selected  in  error  in  the  appropriate  A,  B. 
C.  D.  or  E  column  for  each  question.  Tally  may  be 
accomplished  on  a  separate  sheet.  Destroy  tally  sheets 
immediately  after  posting  totals  on  ATC  Form  668. 

9.  In  the  appropriate  "Number  Wrong  Items"  column 
enter  the  total  missed  by  the  entire  group  for  each  question. 

10.  When  the  form  has  the  required  number  of  groups 
recorded  in  the  "Number  Wrong  Items"  columns,  enter  the 
total  of  the  misses  for  each  question  in  the  "Total  Wrong" 
column. 


11.  Enter  the  difference  between  the  "Number  of  Cases"  (6 
and  the  "Total  Wrong"  (10)  column  in  the  “Total  Right' 
column. 

12.  Enter  all  the  obtainable  percentage  grades  from  I0( 
down  to  below  the  passing  point. 

13.  Indicate  in  the  proper  columns  (I  through  5)  tin 
number  of  students  in  each  class  attaining  each  grade.  Foi 
self-paced  and  multigroup  entry  courses  record  the 
accumulated  number  of  cases  attaining  each  grade. 

14.  When  the  form  has  the  required  number  of  group- 
recorded.  enter  the  total  number  of  students  attaining  eacf 
recorded  grade. 

15.  A  heavy  line  is  drawn  through  this  section  just  unde: 
the  minimum  passing  grade  to  make  the  grade  distrihutioi 
more  meaningful. 

16, 17,  and  18.  Obtain  the  class  average  for  each  group.  I1 
in  group  I.  eight  students  took  the  40 questions  test  there  i- 
a  total  of  320  answers.  By  adding  the  numbers  missed  ir 
column  I  under  the  "Number  Wrong  Items"  section  a  tola 
of  73  misses  is  counted.  (16)  By  subtracting  the  number  oi 
questions  missed  (73)  from  the  total  possible  answers  (320) 
the  result  is  the  total  number  of  correct  answers  (247) 
which  when  divided  by  the  total  possible  (320)  gives  tin 
class  average — 77.2.  (17).  Use  the  same  method  to  obtair 
the  average  difficulty  of  ihe  completed  ATC  Form  668 
(18). 

19.  Fntcr  the  test  item  number.  When  a  lest  consists  ol 
more  than  50  questions,  simply  continue  the  count  or 
another  ATC  Form  668. 


SOTE:  See  sample  completed  ATC  Form  668  in  fi/tun 
A I -2 
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COMPLETION  INSTRUCTIONS  FOR  ATC  FORMS  667  AND  667A 


1.  Type  or  make  all  entries  in  ink 

2.  Before  students  emer  a  block  of  instruction,  or 
immediately  thereafter,  prepare  a  criterion  checklist 
consisting  of  as  many  ATC  Forms  667  or  667A  as  required. 

3.  Enter  the  POI  paragraph  reference  or  the  objectives 
which  is  measured  b\  progress  checks  on  ATC  Form  667  or 
667  A 

4.  Use  ATC  Form  667a  for  FTD  courses  and  courses 
without  a  POI  to  record  objectives. 


5.  Before  administering  a  progress  check,  tell  the  student 
what  is  expected  and  explain  that  the  rating  received 
depends  on  his  or  her  performance. 

6.  Completion  of  blocks  on  ATC  form  667. 

a  Course  number,  block  number,  instructor's  name, 
grade,  and  initials  are  self-explanatory 

b  Under  student's  name  enter  last  name  and  initials  of 
each  student  attending  the  block  of  instruction. 

c  Opposite  the  POI  paragraph  reference  on  ATC  Form 
667.  enter  an  “S"  for  satisfactory,  a  “U"  for  unsatisfactory, 
or  percentage  score. 
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INSTRUCTIONS  TO  THE  EXAMINER-PERFORMANCE  TEST 

EXAMPLE 

DEPARTMENT  OF-  THE  AIR  FORCE 


TESI  II 


LSAF  lechnicu!  I  raining  School 


MEASUREMENT 


COL  RSE  All  courses 

SL  B.IECT  Instructions  to  the  Examiner  (Performance  Test) 

OBJECTIVE  To  provide  standardized  instruction  for  test  administration  and  space  to  include  a  list  of  the  tech  dat 
that  will  he  used 


INSTRl  CTIONS 

1  When  a  controlled  performance  test  is  administered  or  removed  from  the  locked  storage  area  for  any  purpose,  ensui 
that  the  date.  time,  number  sequence  of  tests  issued,  and  the  signature  of  the  person  to  whom  the  test  is  issued  are  entered  o 
the  A  1C  Form  1005.  Test  Control. 

2  L'nsuperv  ised  breaks  are  not  permitted  during  the  performance  test. 

3  Keep  the  testing  area  free  of  distractions  as  much  as  possible  during  examination  Administer  makeup  test  in  a  quie 
area  or  lab  atmosphere  free  from  all  interferences  or  distractions 

4  Student  notes  and  materials  should  he  covered  or  kept  in  a  folder  during  the  examination  period. 

5  Use  of  tech  data  during  test  periods  is  permitted  and  encouraged  when  it  contributes  to  on-the-job  realism  provided  ii 
use  does  not  compromise  the  test 

b  Scratch  p.iper  should  be  furnished  by  the  instructor  if  needed  and  collected  with  the  answer  sheets  and  examinations 

'  Clearly  indicate  time  available  lor  taking  examinations  to  the  students  and  what  is  required  for  satisfactoi- 
accomplishment 

8  Advise  students  if  they  are  authorized  to  ask  questions  or  receive  instructor  assists  during  the  testing  period 

9  If  group  testing  is  required,  ensure  each  student  understands  their  role(s)  as  a  group  participant 

10  Instruct  students  not  to  mark  test  booklets 

11  Tests  are  critiqued  and  each  student  inlormed  of  their  score  as  soon  as  feasible  after  administration 

12  Copies  of  the  test  should  be  returned  temporarily  to  the  students  for  their  review  during  the  critique  Take  care  ti 
reduce  the  possibility  of  test  compromise,  therefore,  during  the  critique  period  use  two  instructors  if  possible  Students  ar- 
not  permitted  to  use  any  written  materials  or  recording  equipment  and  are  not  permitted  to  leave  the  room 

13  Instructors  check  test  booklets  for  markings  before  returning  to  (he  storage  area 

14  The  person  actually  returning  the  examinations  to  the  custodian  certifies  the  examinations  are  clean,  even  though  tha 
person  may  have  obtained  custody  of  the  examinations  through  reissue 

15  List  technical  data  required  during  this  test  administration 
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ATCR  52-3  Attachment  4  25  July  1#®2 


INSTRUCTIONS  TO  THE  EXAMINER— WRITTEN  TEST 
EXAMPLE 


DEPARTMENT  OF  THE  AIR  FORCE 
USAF  Technical  Training  School 


TEST  ID 


MEASUREMENT 

COURSE  All  course;. 

SUBJECI  Instructions  4o  the  Examiner  (Wi itten  Test) 

OBJECT  l\T  To  provide  stundardi7cd  instruction  for  test  administration  and  space  to  include  a  list  of  the  tech  data  that 
will  be  used 

INSTRUCTIONS: 

I  W  hen  a  test  is  administered  or  removed  from  the  locked  storage  area  for  any  purpose,  ensure  that  the  date,  time, 

number  sequence  of  tests  issued,  and  the  signature  of  the  person  to  whom  the  test  is  issued  are  entered  on  the  ATC  Form 

1005.  Test  Control 

2.  Unsupervtscd  breaks  are  not  permitted  during  measurement  test. 

3  Keep  the  classroom  free  of  distractions  such  as  charts,  diagrams,  and  cutaways  during  examination  Administer 

makeup  test  in  a  quiet  classroom  or  lab  atmosphere  free  from  all  interferences  or  distractions. 

4  Student  notes  and  materials  should  be  covered  or  kept  in  a  folder  during  the  examination  period 

5  Use  of  tech  data  during  test  periods  is  permitted  and  encouraged  when  it  contributes  to  on-the-job  realism  provided  its 
use  does  not  compromise  the  test. 

6.  If  possible,  scat  the  students  in  alternate  seats.  In  some  classrooms  it  may  be  possible  to  move  the  chairs  farther  apart 

7.  Scratch  paper  should  be  furnished  by  the  instructor  if  needed  and  collected  with  the  answer  sheets  and  examinations 

8.  Clearly  indicate  time  available  for  taking  examinations  to  the  students  and  minimum  passing  grade. 

9  On  multiple  choice  type  questions  instruct  the  students  to  select  the  best  choice  of  the  answers  listed 

10.  Advise  the  students  to  answer  the  questions  they  are  confident  of  first,  saving  the  more  difficult  questions  until  last 
I  I.  Instruct  students  to  fill  in  all  applicable  headings  on  the  answer  sheet. 

12  Instruct  students  not  to  mark  test  booklets. 

13.  Tests  are  critiqued  and  each  student  informed  of  their  scores  as  soon  as  feasible  after  administration 

14  Scored  answer  sheets  and  copies  of  the  test  should  be  returned  temporarily  to  the  students  for  their  review  during  the 
critique  Take  care  to  reduce  the  possibility  ol  test  compromise:  therefore,  during  the  critique  period  use  two  instructors  il 
possible  Students  are  not  permitted  to  use  any  written  materials  or  recording  equipment  and  are  not  permitted  to  lease  the 
room 

15  Review  test  analysis  data 

16  Instructors  check  test  booklets  lor  markings  before  returning  to  the  storage  area 

17  The  person  actually  returning  the  examinations  to  the  custodian  certifies  the  examinations  are  clean,  even  though  that 
person  mav  have  obtained  custody  of  the  examinations  through  reissue 

18  Annotate  a  copv  of  these  instructions  lor  each  writfen  test  and  issued  with  the  lest 

19  List  technical  data  required  during  this  tesi  administration 


APPENDIX  C:  STUDENT  RECORD  OF  TRAINING 
(ATC  FORM  156) 
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APPE.NO I K  D:  AVERAGE  DIFFICULTY  RATINGS  BY  HIGH  SCHOOL  COURSE 


# 

Title 

Average 

Rating 

# 

Title 

Average 

Rating 

I 

Speech 

2.4 

22 

Hydrology 

3.7 

L 

Algebra 

4.1 

23 

Hygiene 

1.5 

3 

E  lectronics 

3.4 

24 

English 

3.8 

4 

Biol ogy 

4.0 

25 

Industrial  Arts 

1.8 

5 

Accounting 

3.0 

26 

Logarithms 

4.0 

6 

Business  Math 

2.6 

27 

General  Business 

2.4 

7 

Chemistry 

4.6 

28 

Business 

Administration 

3.0 

8 

Conroercial  Art 

2.1 

29 

Driver  Training 

1.5 

9 

Mechanical  Drawing 

2.4 

30 

Geography 

3.1 

10 

Genera)  Science 

3.0 

3) 

Management 

2.9 

!  1 

Geometry 

4.2 

32 

Marketing 

2.7 

12 

Journal  ism 

3.4 

33 

Mechanical  Theory 

3.4 

t3 

Zoology 

3.8 

34 

Merchandising 

2.2 

14 

Physics 

4.9 

35 

Home  Economics 

1.7 

IS 

Psychology 

3.7 

36 

Business  Machine 
Operator 

1.5 

16 

Projection  Equipment 

1.6 

37 

Statistics 

4.2 

17 

Radio  Repair 

2.5 

38 

General  Math 

2.6 

18 

Trigonometry 

4.4 

39 

Shop  Math 

2.1 

19 

Use  of  Blueprints 

2.8 

40 

Vocational  Guidance 

2.4 

20 

Anatomy 

3.9 

41 

Music 

<r  .2 

21 

Auto  Repair 

2.6 

42 

Agriculture 

2.3 
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APPENDIX  H:  INTERCORRELATIONS  OF  STUDENT  INPUT,  COURSE  CONTENT 
AND  TRAINING  OUTCOME  VARIABLES  IN  THE  TOTAL  SAMPLE 
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